diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7d247bc..67d8861 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -200,11 +200,16 @@ start on at all (`checkExternalURL`, `checkPort`, `checkBrowserHost`, Tier A string in `internal/config/normalize.go`, which `TestContractAddressShapes` holds the render to and test/crd's `TestAddressShapesFollowTheEngine` the API's rules on a Company's and a -class's addresses, at the shapes they try) — with +class's addresses, at the shapes they try; and, for the address a pod hands +the engine's sandboxes, `ParseBoxURL` and its helpers in +`internal/sandbox/boxurl.go`, through which the boot reads each such +variable (`ReadBoxURLs` in `internal/engine/boxurl.go`), and which +`TestContractBoxURLShapes` holds the render to by asking `crewlet llm doctor`) — with every operator file that copies it and a digest of the function as it was read. Those files are the renderer's; for the addresses, also the Company and EngineClass API types whose rules refuse them, the controller's message -on a Company held back for one, and the [exposure](docs/guides/exposure.md) +on a Company held back for one and its choice of the addresses a pod hands +sandboxes, and the [exposure](docs/guides/exposure.md) and [troubleshooting](docs/guides/troubleshooting.md) guides that state them; and, for how an agent-mode run's CLI signs in inside the remote sandbox, also the LLMProvider API's rule on a CLI of mode `Agent`, the @@ -232,9 +237,62 @@ a seat pod's boot, which the controller's startup windows add up EngineClass API floors `data.startupSeconds` at, and the runtime and troubleshooting pages quote, and one step the windows leave out, past the recovery boundary: the boot's read of the company's secrets (`bootSecrets`), -which waits out a store that does not answer. Which steps a boot runs, and in -what order, is `engine.New`'s, which is not read, since it changes with most -of the engine's work. The digest (`enginetest.Fingerprint`) is taken over the +which waits out a store that does not answer. It names, too, the one wait +the engine bounds by no time, which the windows count as a boot meets it: +the agreement of the object store's fleet record, bounded by a number of +attempts (`casRetry`), at the one attempt a boot meets. How long a read may +wait is read down to the function that bounds it, for every read the engine +asks again while nobody answers it (`hedge` and `ReadBoundedWith` in +`internal/jsprovision/hedge.go`): a leader read, the pages of a pass's key +index and the broker's limit read are held to the client's own five seconds +where their caller set no deadline (`(*Leader).get`, `NewLeader`, `ask`, +`askTimed`, `page`, `sorted` and `(*Leader).Subjects` in `internal/jsapi`, +`keysUnder` in `internal/coord/kv/walk.go`, `(*Queue).budget` in +`internal/queue/jetstream/budget.go`), and a read of a state log's state or +record to the thirty seconds of its timing's `Lookup` (`ReadWith`, with the +queue's timing, `(*Queue).provisioning` and `Clustered.Timing`), which a +boot's read waits out, its context having no deadline. Each read of one key +the windows count is one such leader read, of the key's newest message +(`(*leaderReader).last` in `internal/coord/kv/walk.go` and `(*Leader).Last` +in `internal/jsapi`), whether it reads the key's live value +(`(*FleetStore).get` in `internal/coord/kv/fleet.go`, `getLatest` and +`(*leaderReader).live`), is a round of a create (`(*FleetStore).create`, and +`createKey` in `internal/coord/kv/markers.go`) or is a listing's read back. +The write each round of that create makes before its read is read down to +its bound too: it is a settled update, which waits out the leader's answer +that another write to the key is still in flight, no longer than the +client's own five seconds where its caller set no deadline (`settled`, +`settledBucket.Update` and `settledBucket.settle` in +`internal/coord/kv/settle.go`, and `Decide` in +`internal/jsinflight/jsinflight.go`). That the bucket it writes is settled +is not read: Crewlet's own +`TestAConditionalWriteIsAnsweredByWhatTheLeaderDecided` requires it. +How many registrations a step makes is read down to the queue, for every +step that registers what a node answers: the steer answerer and the fleet +broker register in their step's own code (`armSteer`, `serveFleetBroker`); +the history answerer, the inbox relay and the presence answerer each +register one subject in the function their step calls (`Serve` in +`internal/eventfan/fanwire.go`, `(*inboxRelay).serve` in +`internal/engine/inboxrelay.go` and `(*Hub).Start` in +`internal/presence/hub.go`); the estate service registers each of its lanes +in turn (`Serve` and `lanes` in `internal/estate`), two on a data pod; and +every registration is one flush (`(*Queue).Serve` and `flush` in +`internal/queue/jetstream/scatter.go`). So a registration added below a +function whose own code does not change still fails the test. +The listing the windows count, the join's of the fleet's positions, is read +down the same way, to the certified listing it is and to the functions that +bound that listing's waits: the positions register's walk of one class +(`(*FleetStore).eachPositionKey` in `internal/coord/kv/positions.go`), a +fleet bucket's filtered walk (`(*FleetStore).eachUnder` in +`internal/coord/kv/fleet.go`, `eachEntryUnder` and `newLeaderReader` in +`internal/coord/kv/walk.go`), and the pass a certified listing is, with its +read back of a key the pass did not deliver live (`listUnder`, `beginPass`, +`(*pass).walk`, `passOver`, `passIdle`, `(*pass).stop`, `(*pass).certify`, +`certify`, `(*pass).close` and `(*leaderReader).last` in the same file, and +`(*Leader).Last` in `internal/jsapi`). So a request added below that listing +fails it too. +Which steps a boot runs, and in what order, is `engine.New`'s, which is not +read, since it changes with most of the engine's work. The digest (`enginetest.Fingerprint`) is taken over the function, or over the one spec that declares a constant, as `go/printer` lays it out without its comments or the source's line breaks, so a comment or a rewrapped line leaves it alone and a changed statement, value, spread diff --git a/SECURITY.md b/SECURITY.md index 431241e..580e861 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -78,15 +78,21 @@ breaks one of them is a vulnerability and should be reported as above. reach them, unless the EngineClass routes the whole API (`exposure.api`) through a Gateway or an Ingress of your own, which should be internal. Your CNI must enforce NetworkPolicy for that layer to exist. -- **The data pods' leaf listener is unauthenticated.** While a company may - run seat pools, its data pods listen on port 7422 for the seat pods to - join the event stream, with no authentication and no TLS; whoever connects - reads and writes the company's whole event stream. The NetworkPolicy +- **The data pods' leaf listener admits only the company's keys, and + encrypts nothing.** While a company may run seat pools, its data pods + listen on port 7422 for the seat pods to join the event stream. Crewlet's + broker admits a link there only when it presents a key derived from the + company's keyring, so whoever can read a Secret that carries it can join: + `-keyring`, `-bootstrap` or any + `-bootstrap--`. Whoever joins reads and writes the + company's whole event stream; the link carries no TLS, so whoever can watch the pod network + between the pods reads what crosses it. The NetworkPolicy `-broker` admits the port only from pods carrying the company's - seat-pod labels, so the listener is closed to other namespaces only where - the CNI enforces NetworkPolicy, and within a namespace only where nobody - but the operator creates pods: one company per namespace, or a namespace - no tenant can create pods in + seat-pod labels, so other pods cannot reach the listener at all, but only + where the CNI enforces NetworkPolicy. Within a namespace neither holds + against whoever can create pods there, since such a pod can carry the + labels and mount any of those Secrets: one company per namespace, or a + namespace no tenant can create pods in ([Network policy](docs/concepts/runtime.md#network-policy)). - **The manager's metrics endpoint requires authentication.** It is served over HTTPS and every request is authorised against the Kubernetes API, diff --git a/api/v1alpha1/common_types.go b/api/v1alpha1/common_types.go index 550588c..d369fb2 100644 --- a/api/v1alpha1/common_types.go +++ b/api/v1alpha1/common_types.go @@ -152,13 +152,19 @@ type SeatCredentialSource struct { } // SeatEnvVar is a named credential of a seat delivered as an environment -// variable: from a Secret, or provisioned by the engine. +// variable of its sandbox runs (a Seat's spec.sandbox.env): from a Secret, or +// provisioned by the engine. type SeatEnvVar struct { - // name is the variable's name. + // name is the variable's name. It may not be one Crewlet sets in a coding + // run itself, which Crewlet refuses in every company document: + // CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + // CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + // ReservedSandboxEnv). // +required // +kubebuilder:validation:MinLength=1 // +kubebuilder:validation:MaxLength=128 // +kubebuilder:validation:Pattern=`^[A-Za-z_][A-Za-z0-9_]*$` + // +kubebuilder:validation:XValidation:rule="!(self in ['CREWLET_GIT_TOKEN', 'CREWLET_GIT_URL', 'CREWLET_GIT_TRAILER_SEAT', 'CREWLET_GIT_TRAILER_RUN', 'CREWLET_GIT_TRAILER_WORK_ITEM'])",message="CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM are the engine's: Crewlet sets them in a coding run itself, as the run's credential on the company's own code host, the base its clones and pushes reach and the values its commit hook writes into every commit's trailers, and refuses a company document that sets one, since a value here would break the run's clones and pushes or make a commit's trailers disagree with its push's record; remove it" Name string `json:"name,omitempty"` // valueFrom is where the value comes from. @@ -167,9 +173,9 @@ type SeatEnvVar struct { } // SecretEnvVar is a named credential delivered as an environment variable, -// such as a sandbox setup step's token or a coding CLI's model key. Crewlet -// treats every value in the positions that use this type as a credential, so -// a literal value is not accepted: the value always comes from a Secret. +// such as a coding CLI's model key. Crewlet treats every value in the +// positions that use this type as a credential, so a literal value is not +// accepted: the value always comes from a Secret. type SecretEnvVar struct { // name is the variable's name. // +required diff --git a/api/v1alpha1/company_types.go b/api/v1alpha1/company_types.go index edcc7c1..7bb0869 100644 --- a/api/v1alpha1/company_types.go +++ b/api/v1alpha1/company_types.go @@ -20,32 +20,14 @@ const ( TrackerNone TrackerBackend = "None" ) -// WorkletSettings is the policy of Worklet. It maps onto Crewlet's -// tracker.worklet. -// +kubebuilder:validation:MinProperties=1 -type WorkletSettings struct { - // inboxRetentionDays is how long a person's inbox keeps a row; the - // history the row points at is kept regardless. Unset takes Crewlet's - // default of 365. Crewlet's `inbox_retention_days`. - // +optional - // +kubebuilder:validation:Minimum=30 - // +kubebuilder:validation:Maximum=3650 - InboxRetentionDays int32 `json:"inboxRetentionDays,omitempty"` -} - // TrackerSettings is which work tracker a company runs. It maps onto // Crewlet's tracker block. -// +kubebuilder:validation:XValidation:rule="!has(self.worklet) || self.backend == 'Worklet'",message="worklet applies to backend Worklet only" // +kubebuilder:validation:MinProperties=1 type TrackerSettings struct { // backend is which tracker the company runs. // +optional // +default="Worklet" Backend TrackerBackend `json:"backend,omitempty"` - - // worklet is the policy of Worklet, for backend Worklet. - // +optional - Worklet WorkletSettings `json:"worklet,omitzero"` } // KnowledgeBackend is which knowledge base a company runs. It maps onto @@ -111,6 +93,29 @@ type KnowledgeSettings struct { Vectors *bool `json:"vectors,omitempty"` } +// InboxSettings is the company's policy over a person's inbox: the notices +// the engine writes to each person its work concerns. It maps onto +// Crewlet's inbox block. +// +// It is the company's rather than its tracker's, and applies whatever +// tracker the company runs: a person's inbox holds the notices of every part +// of the engine that writes them, Worklet's among them, and one horizon ages +// them all. +// +kubebuilder:validation:MinProperties=1 +type InboxSettings struct { + // retentionSeconds is how long a person's inbox keeps a notice; the + // history the notice points at is kept regardless. Crewlet keeps the + // horizon in whole days, so it is a whole number of days in seconds, + // from 30 days (2592000) to 3650 days (315360000), such as 7776000 for + // 90 days. Unset takes Crewlet's default of 365 days. Crewlet's + // `inbox.retention_days`, which is this divided by 86400. + // +optional + // +kubebuilder:validation:Minimum=2592000 + // +kubebuilder:validation:Maximum=315360000 + // +kubebuilder:validation:XValidation:rule="self % 86400 == 0",message="retentionSeconds must be whole days, a multiple of 86400 such as 7776000 for 90 days: Crewlet keeps a person's inbox for whole days" + RetentionSeconds int32 `json:"retentionSeconds,omitempty"` +} + // LLMProviderKind is the kind of model provider a reference names. // +kubebuilder:validation:Enum=LLMProvider;PlatformLLMProvider type LLMProviderKind string @@ -300,6 +305,34 @@ type SandboxFile struct { ValueFrom SecretValueSource `json:"valueFrom,omitzero"` } +// SandboxEnvVar has SecretEnvVar's shape and is a type of its own for the rule +// on its name: Crewlet reserves the variables it sets in a coding run itself +// in each env a run's environment is assembled from (a setup step's, and a +// Seat's spec.sandbox.env, whose SeatEnvVar holds the same rule), and nowhere +// else, so an LLMProvider's env, which SecretEnvVar types, may hold them. + +// SandboxEnvVar is a variable a sandbox setup step merges into the +// environment of its commands and of a coding run. Crewlet treats every value +// there as a credential, so a literal value is not accepted: the value always +// comes from a Secret. +type SandboxEnvVar struct { + // name is the variable's name. It may not be one Crewlet sets in a coding + // run itself, which Crewlet refuses in every company document: + // CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + // CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + // ReservedSandboxEnv). + // +required + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=128 + // +kubebuilder:validation:Pattern=`^[A-Za-z_][A-Za-z0-9_]*$` + // +kubebuilder:validation:XValidation:rule="!(self in ['CREWLET_GIT_TOKEN', 'CREWLET_GIT_URL', 'CREWLET_GIT_TRAILER_SEAT', 'CREWLET_GIT_TRAILER_RUN', 'CREWLET_GIT_TRAILER_WORK_ITEM'])",message="CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM are the engine's: Crewlet sets them in a coding run itself, as the run's credential on the company's own code host, the base its clones and pushes reach and the values its commit hook writes into every commit's trailers, and refuses a company document that sets one, since a value here would break the run's clones and pushes or make a commit's trailers disagree with its push's record; remove it" + Name string `json:"name,omitempty"` + + // valueFrom is the Secret key the value is read from. + // +required + ValueFrom SecretValueSource `json:"valueFrom,omitzero"` +} + // The optionalKeys rule of SandboxSetupStep compares every entry with every // variable of env and every path of files, and the API server sums its // estimated cost over every step a Company's or a Seat's spec.sandbox.setup @@ -356,12 +389,17 @@ type SandboxSetupStep struct { Commands []string `json:"commands,omitempty"` // env is merged into the environment of the commands and of the run. - // Crewlet's `env`, a credential position. + // It may not set a variable Crewlet sets in a coding run itself, which + // Crewlet refuses in every company document, as a Seat's + // spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL, + // CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and + // CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's + // `env`, a credential position. // +optional // +listType=map // +listMapKey=name // +kubebuilder:validation:MaxItems=32 - Env []SecretEnvVar `json:"env,omitempty"` + Env []SandboxEnvVar `json:"env,omitempty"` // optionalKeys names variables of env, and paths of files, whose value may go // missing. Crewlet refuses to start a run_sandbox coding run whose environment @@ -685,12 +723,26 @@ type CompanyExposure struct { // is Go's net/url as the API server was built and is run: it takes such // a zone, and under Go's GODEBUG setting urlstrictcolons=0 // https://h.example:80:80 too. + // + // The engine also hands it to its sandboxes, as its tool bridge's address + // (CREWLET_MCP_BRIDGE_URL) while an LLMProvider of the company runs in + // mode Agent, and as its telemetry receiver's + // (CREWLET_SANDBOX_OTEL_RECEIVER_URL) while the class sends telemetry to + // a collector, and refuses to start on one the URL parser of a sandbox's + // coding agent refuses, the WHATWG URL Standard's. So its host may not + // hold an IPv6 zone or another '%', a '<', a '>' or a ']' outside an + // IPv6 address's brackets, and a host whose last label is a number must + // be an IPv4 address written as four decimal octets: that parser reads + // such a host as an address, and Crewlet refuses one that is none. // +optional // +kubebuilder:validation:MinLength=1 // +kubebuilder:validation:MaxLength=2048 // +kubebuilder:validation:XValidation:rule="isURL(self) && self.startsWith('https://') && url(self).getHostname() != ''",message="publicBaseURL must be an absolute URL starting with https://" // +kubebuilder:validation:XValidation:rule="!isURL(self) || !self.matches('^https?://[^/]*:(0+|0*([1-9][0-9]{5,}|[7-9][0-9]{4}|6[6-9][0-9]{3}|65[6-9][0-9]{2}|655[4-9][0-9]|6553[6-9]))(/|$)')",message="publicBaseURL's port must be 1 to 65535: Crewlet refuses to start on any other; leave the port out for https's own, 443" // +kubebuilder:validation:XValidation:rule="!self.matches('^[^:/?#]+://') || self.matches('^[^:/?#]+://([^/?#]*@)?([[](([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|([0-9A-Fa-f]{1,4}:){1,7}:|:(:[0-9A-Fa-f]{1,4}){1,7}|::|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){1,6}|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){1,5}|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){1,4}|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){1,3}|([0-9A-Fa-f]{1,4}:){5}(:[0-9A-Fa-f]{1,4}){1,2}|([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){6}|:(:[0-9A-Fa-f]{1,4}){0,5}:|([0-9A-Fa-f]{1,4}:){5}:|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){0,4}:|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){0,3}:|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){0,2}:|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){0,1}:)(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])([.](25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])){3})(%25[^]/?#]+)?[]]|[^/?#:[]*)(:[0-9]*)?([/?#]|$)')",message="publicBaseURL's host must be a name, an IPv4 address or an IPv6 address in brackets, and at most one ':' and a port may follow it: Crewlet refuses to start on any other address, or starts on one no vendor reaches (write https://[2001:db8::1]:8443, not https://2001:db8::1:8443)" + // +kubebuilder:validation:XValidation:rule="!isURL(self) || !self.matches('^https?://[^/]*%')",message="publicBaseURL's host may not hold a '%': the URL parser of a sandbox's coding agent refuses an IPv6 zone (%25eth0) and a character no host may hold, and Crewlet refuses to start when it hands a sandbox such an address; write the host as it is, and an IPv6 address without a zone" + // +kubebuilder:validation:XValidation:rule="!isURL(self) || !self.matches('^https?://[^/[]*[]<>]')",message="publicBaseURL's host may not hold a '<', a '>' or a ']': the URL parser of a sandbox's coding agent refuses each in a host name, as it does an unfilled placeholder such as https://, and Crewlet refuses to start when it hands a sandbox such an address" + // +kubebuilder:validation:XValidation:rule="!isURL(self) || !self.matches('^https?://([^/:]*[.])?([0-9]+|0[xX][0-9a-fA-F]*)[.]?(:[0-9]*)?(/|$)') || self.matches('^https?://((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])[.]){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(:[0-9]*)?(/|$)')",message="publicBaseURL's host ends in a number, which the URL parser of a sandbox's coding agent reads as an IPv4 address, and Crewlet refuses to start when it hands a sandbox one that is none: write an IPv4 address as four decimal octets, such as 192.0.2.10, and a host name whose last label is not a number" // +kubebuilder:validation:XValidation:rule="!self.matches('^[^:/?#]+://[^/?#]*@')",message="publicBaseURL may not hold a user name or password: a credential is named through a Secret, never written into an address" // +kubebuilder:validation:XValidation:rule="!self.contains('?') && !self.contains('#')",message="publicBaseURL may not have a query or a fragment: Crewlet builds every webhook and sandbox address by appending a path to it" // +kubebuilder:validation:XValidation:rule="!self.matches('[$][{][A-Za-z_][A-Za-z0-9_]*[}]')",message="publicBaseURL may not hold a ${NAME} reference: the operator refuses every one it did not write, since in many fields the engine resolves one from its own secret store and environment, which hold credentials that are not this object's to name" @@ -990,6 +1042,10 @@ type CompanySpec struct { // +optional Knowledge KnowledgeSettings `json:"knowledge,omitzero"` + // inbox is the company's policy over a person's inbox. + // +optional + Inbox InboxSettings `json:"inbox,omitzero"` + // tokenBudget caps the tokens all seats together may spend per window. // Crewlet's `token_budget`. // +optional @@ -1022,17 +1078,24 @@ type CompanySpec struct { // apply, and a refusal is reported on the Company. // // It may not set what another field or object of this API owns: name, - // mission, vision, policies, timezone, tracker, knowledge, token_budget, - // providers, integrations, mcp_servers, roles or units. Every position - // Crewlet holds a credential in lives under one of those keys, so - // engineConfig can never carry a credential: the API server refuses a - // Company whose engineConfig names one of them, before anything is - // stored. + // mission, vision, policies, timezone, tracker, knowledge, inbox, + // token_budget, providers, integrations, mcp_servers, roles or units. + // Every position Crewlet holds a credential in lives under one of those + // keys, so engineConfig can never carry a credential: the API server + // refuses a Company whose engineConfig names one of them, before anything + // is stored. + // + // Nor may it set code, Crewlet's code host, which keeps the company's git + // repositories in the engine and which v1alpha1 does not offer: the host + // serves them only from a node with the git role and a cache of them on + // its volume, and no pod the operator runs has either, so a document + // that set it would declare repositories nothing can clone or push. // +optional // +listType=map // +listMapKey=name // +kubebuilder:validation:MaxItems=64 - // +kubebuilder:validation:XValidation:rule="self.all(e, !(e.name in ['name', 'mission', 'vision', 'policies', 'timezone', 'tracker', 'knowledge', 'token_budget', 'providers', 'integrations', 'mcp_servers', 'roles', 'units']))",message="engineConfig may not set name, mission, vision, policies, timezone, tracker, knowledge, token_budget, providers, integrations, mcp_servers, roles or units: those are fields and objects of this API, and the only blocks that hold credentials" + // +kubebuilder:validation:XValidation:rule="self.all(e, !(e.name in ['name', 'mission', 'vision', 'policies', 'timezone', 'tracker', 'knowledge', 'inbox', 'token_budget', 'providers', 'integrations', 'mcp_servers', 'roles', 'units']))",message="engineConfig may not set name, mission, vision, policies, timezone, tracker, knowledge, inbox, token_budget, providers, integrations, mcp_servers, roles or units: those are fields and objects of this API, and the only blocks that hold credentials" + // +kubebuilder:validation:XValidation:rule="self.all(e, e.name != 'code')",message="engineConfig may not set code: Crewlet's code host is not offered in v1alpha1, since no pod the operator runs has the git role or the cache that serves a company's repositories, which nothing could clone or push; remove the entry" EngineConfig []EngineConfigEntry `json:"engineConfig,omitempty"` // config is how the operator manages the company's document. diff --git a/api/v1alpha1/engineclass_types.go b/api/v1alpha1/engineclass_types.go index ca02b4e..6dbcfb9 100644 --- a/api/v1alpha1/engineclass_types.go +++ b/api/v1alpha1/engineclass_types.go @@ -64,16 +64,18 @@ type DataStorage struct { // stream.store_max_bytes: the stream reserves what a node keeps there up // front, and a fresh node refuses to boot on a share that cannot hold it // (Crewlet's engine.BrokerFloor, and its deployment guide's "minimum room - // for a fresh node"). Its seven state logs' ceilings at their smallest - // come to 6979321856 bytes, so a volume must be at least 9Gi, whose share - // of 6.75Gi holds them. With objects.backend NATS, the default, the object - // store's bucket reserves a quarter of the share beside them, 9305762474 - // bytes in all, so such a class must give at least 12Gi, whose share is - // 9Gi. That is the least that boots: the mailboxes and coordination - // buckets grow into what is left, so give a company that runs for long - // more. The stream's ceiling is 64Ti (config.StoreMaxBytesCeiling, a - // guard against a unit mistake), so a volume larger than about 85Ti gives - // the stream 64Ti and the rest to the store and its snapshots. + // for a fresh node"). Its eight state logs' ceilings at their smallest + // (the code host's log among them, which a node reserves whether or not + // the company hosts code) come to 8053063680 bytes, so a volume must be + // at least 10Gi, whose share of 7.5Gi holds them. With objects.backend + // NATS, the default, the object store's bucket reserves a quarter of the + // share beside them, 10737418239 bytes in all, so such a class must give + // at least 14Gi, whose share is 10.5Gi. That is the least that boots: the + // mailboxes and coordination buckets grow into what is left, so give a + // company that runs for long more. The stream's ceiling is 64Ti + // (config.StoreMaxBytesCeiling, a guard against a unit mistake), so a + // volume larger than about 85Ti gives the stream 64Ti and the rest to the + // store and its snapshots. // // It applies to the data volumes of companies created after it is set. // An existing company's volumes keep the size they were created with: @@ -82,8 +84,14 @@ type DataStorage struct { // decision about one company's storage class and data, not the class's. // Its owner expands the company's claim instead, and once the expansion // lands the operator sizes the stream from what the claim then has. + // + // A class stored below the floor, under an older one, keeps its size, + // and every Company created under it gets a volume no fresh node boots + // on, so raise it. Until it is raised the API refuses any change to the + // class's spec, or, with objects.backend S3, a change of its size that + // leaves it below the floor. // +required - // +kubebuilder:validation:XValidation:rule="!quantity(string(self)).isLessThan(quantity('9Gi'))",message="size must be at least 9Gi: three quarters of the volume is the engine's event stream, and a fresh node of Crewlet reserves at least 6979321856 bytes of it for its state logs" + // +kubebuilder:validation:XValidation:rule="!quantity(string(self)).isLessThan(quantity('10Gi'))",message="size must be at least 10Gi: three quarters of the volume is the engine's event stream, and a fresh node of Crewlet reserves at least 8053063680 bytes of it for its state logs" Size *resource.Quantity `json:"size,omitempty"` } @@ -123,7 +131,7 @@ type DataPodSettings struct { // that hung. // // It is at least every bound Crewlet puts on a data pod's boot, laid end - // to end, which a window may not cut short: 1636 seconds, or 1776 with + // to end, which a window may not cut short: 1461 seconds, or 1601 with // objects.backend S3, whose bucket every boot probes. Those are its waits // for its own broker and store (provisioning the event stream, the // coordination buckets, the object store and the state logs, two minutes @@ -131,17 +139,20 @@ type DataPodSettings struct { // around them), never for a peer or a seat pod. One of them grows with // what the node holds and is left to the recovery's share, as the // recovery is: the claim of each batch of a seat pod's events the node - // kept and could not settle, 15 s a batch. Unset derives it from the - // data volume: that floor plus six seconds per GiB of the volume. That is - // four times the second and a half per GiB Crewlet measured recovering - // after a crash on a fast local disk, for volumes that are + // kept and could not settle, 15 s a batch. And a read of a state log's + // state or record is counted at the 30 s Crewlet asks it for while the + // broker answers none, past which it gives the read up, so no broker + // that never answers holds a boot past the window. Unset derives it from + // the data volume: that floor plus six seconds per GiB of the volume. + // That is four times the second and a half per GiB Crewlet measured + // recovering after a crash on a fast local disk, for volumes that are // network-attached; set this for disks slower than that. Nothing the // company configures counts: the engine answers /health once it has // recovered, and only then reads the company's secrets, starts its tool // servers and restores its CLIs' logins. // Changing it changes the data pods' template, which restarts them. // +optional - // +kubebuilder:validation:Minimum=1636 + // +kubebuilder:validation:Minimum=1461 // +kubebuilder:validation:Maximum=86400 StartupSeconds int32 `json:"startupSeconds,omitempty"` @@ -558,12 +569,22 @@ type WebhookExposure struct { // {{name}} and {{namespace}} joined by a '.', which neither can hold, so // that no two companies share a host name: with a '-' between them, // team-a/b-web and team-a-b/web would. + // + // The engine hands the address to its sandboxes too, as its tool + // bridge's and its telemetry receiver's, and refuses to start on one the + // URL parser of a sandbox's coding agent reads as an IPv4 address that + // is none, as it does a host whose last label is a number. So that last + // label may not be one ({{uid}}.123, {{uid}}.0x1f). One a variable fills + // is a number only for some Companies, as api.hostname says, and the + // operator holds that Company's engine back, with RuntimeReady=False + // AddressRefused, while it hands the address to a sandbox. // +optional // +kubebuilder:validation:MinLength=1 // +kubebuilder:validation:MaxLength=253 // +kubebuilder:validation:XValidation:rule="!self.replace('{{namespace}}', '').replace('{{name}}', '').replace('{{uid}}', '').contains('{' + '{')",message="hostname may use only the variables {{namespace}}, {{name}} and {{uid}}" // +kubebuilder:validation:XValidation:rule="self.contains('{{uid}}') || self.contains('{{name}}.{{namespace}}') || self.contains('{{namespace}}.{{name}}')",message="hostname must contain {{uid}}, or {{name}} and {{namespace}} joined by a '.', so that no two companies share a host name: names and namespaces hold '-', so {{namespace}}-{{name}} is team-a-b-web for both team-a/b-web and team-a-b/web" // +kubebuilder:validation:XValidation:rule="self.replace('{{namespace}}', 'n').replace('{{name}}', 'n').replace('{{uid}}', 'u').matches('^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$')",message="hostname must be a lowercase DNS name once its variables are replaced" + // +kubebuilder:validation:XValidation:rule="!self.matches('(^|[.])([0-9]+|0x[0-9a-f]*)$')",message="hostname may not end in a label that is a number, such as {{uid}}.123 or {{uid}}.0x1f: a sandbox's URL parser reads the host as an IPv4 address, and Crewlet refuses to start when it hands a sandbox one that is none; end it in a label that is not one, such as a domain of your own" Hostname string `json:"hostname,omitempty"` } @@ -636,7 +657,7 @@ type APIExposure struct { // refuses to start on a browser's address whose host ends in a label // that is a number: a browser reads such a host as an IPv4 address and // rewrites it before it sends it. So its last label may not be one - // ({{uid}}.123, {{uid}}.0x1f), where webhooks.hostname's may. A last + // ({{uid}}.123, {{uid}}.0x1f), as webhooks.hostname's may not. A last // label a variable fills is a number only for some Companies: one whose // namespace is a number (4711, 0x1f) under {{name}}.{{namespace}}, or // whose namespace or name makes one with the template's own characters @@ -1209,8 +1230,8 @@ type SignInSettings struct { // a company's object store cannot move under them, since Crewlet refuses to // start a node that names another store than the one it recorded, so the // backend a volume and a window were sized for is the one it runs with. -// +kubebuilder:validation:XValidation:rule="(has(self.objects) && has(self.objects.backend) && self.objects.backend == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('12Gi'))",message="data.storage.size must be at least 12Gi with objects.backend NATS: three quarters of the volume is the engine's event stream, and a fresh node of Crewlet reserves at least 9305762474 bytes of it for its state logs and the object store's bucket" -// +kubebuilder:validation:XValidation:rule="!has(self.data.startupSeconds) || !has(self.objects) || !has(self.objects.backend) || self.objects.backend != 'S3' || self.data.startupSeconds >= 1776",message="data.startupSeconds must be at least 1776 with objects.backend S3: every boot of a data pod probes the bucket, and Crewlet's bounds on the boot then add up to 1776 seconds, which a shorter window can cut short" +// +kubebuilder:validation:XValidation:rule="(has(self.objects) && has(self.objects.backend) && self.objects.backend == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('14Gi'))",message="data.storage.size must be at least 14Gi with objects.backend NATS: three quarters of the volume is the engine's event stream, and a fresh node of Crewlet reserves at least 10737418239 bytes of it for its state logs and the object store's bucket" +// +kubebuilder:validation:XValidation:rule="!has(self.data.startupSeconds) || !has(self.objects) || !has(self.objects.backend) || self.objects.backend != 'S3' || self.data.startupSeconds >= 1601",message="data.startupSeconds must be at least 1601 with objects.backend S3: every boot of a data pod probes the bucket, and Crewlet's bounds on the boot then add up to 1601 seconds, which a shorter window can cut short" // +kubebuilder:validation:XValidation:rule="self.networkPolicy.mode != 'Isolated' || ((!has(self.exposure.webhooks.ingress) || has(self.exposure.webhooks.ingress.controllerNamespaces)) && (!has(self.exposure.api.ingress) || has(self.exposure.api.ingress.controllerNamespaces)) && (!has(self.exposure.artifacts) || !has(self.exposure.artifacts.ingress) || has(self.exposure.artifacts.ingress.controllerNamespaces)))",message="networkPolicy Isolated with an Ingress needs ingress.controllerNamespaces: without them the policy refuses everything the Ingress forwards" // +kubebuilder:validation:XValidation:rule="!has(self.networkPolicy) || !has(self.networkPolicy.mode) || self.networkPolicy.mode != 'Isolated' || !has(self.exposure) || !has(self.exposure.webhooks) || !has(self.exposure.webhooks.mode) || self.exposure.webhooks.mode != 'External' || has(self.exposure.webhooks.external)",message="networkPolicy Isolated with External webhooks needs webhooks.external.forwarders: without them the policy refuses everything forwarded to the public port" // +kubebuilder:validation:XValidation:rule="!has(self.exposure) || !has(self.exposure.webhooks) || !has(self.exposure.webhooks.external) || (has(self.networkPolicy) && has(self.networkPolicy.mode) && self.networkPolicy.mode == 'Isolated')",message="webhooks.external.forwarders apply to networkPolicy mode Isolated only: no other mode has a policy to admit them to" diff --git a/api/v1alpha1/seat_types.go b/api/v1alpha1/seat_types.go index 16a107e..a0a5999 100644 --- a/api/v1alpha1/seat_types.go +++ b/api/v1alpha1/seat_types.go @@ -168,8 +168,16 @@ type SeatSandbox struct { // endpoint of the provider the code work runs on out of the box, unless // a run goes without its value (optionalKeys), when the box is given // them as it is with no sign-in; and a variable named as one the - // provider would set wins over it. Crewlet's `env`, a credential - // position. + // provider would set wins over it. It may not set a variable Crewlet + // sets in a coding run itself, which Crewlet refuses in every company + // document, and nor may a setup step's env: CREWLET_GIT_TOKEN and + // CREWLET_GIT_URL, a run's credential on the company's own code host and + // the base its clones and pushes reach, and CREWLET_GIT_TRAILER_SEAT, + // CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM, the values + // its commit hook writes into every commit's trailers (Crewlet's + // ReservedSandboxEnv). No run of a company the operator runs is given + // them, since the operator does not offer the code host, but the names + // are the engine's all the same. Crewlet's `env`, a credential position. // +optional // +listType=map // +listMapKey=name @@ -240,19 +248,39 @@ type SlackSeatIntegration struct { CredentialsSecretRef SecretReference `json:"credentialsSecretRef,omitzero"` } -// GitHubAccessTier is how much a seat may do in a repository. It maps onto -// Crewlet's role integrations.github.tier (ReadOnly is `read_only`, Review is -// `review`, FullAccess is `full_access`). +// CodeAccessTier is how much a seat may do in a repository. It maps onto +// Crewlet's role code.access (ReadOnly is `read_only`, Review is `review`, +// FullAccess is `full_access`). // +kubebuilder:validation:Enum=ReadOnly;Review;FullAccess -type GitHubAccessTier string +type CodeAccessTier string -// The GitHub access tiers, least first. +// The code access tiers, least first. const ( - GitHubAccessReadOnly GitHubAccessTier = "ReadOnly" - GitHubAccessReview GitHubAccessTier = "Review" - GitHubAccessFullAccess GitHubAccessTier = "FullAccess" + CodeAccessReadOnly CodeAccessTier = "ReadOnly" + CodeAccessReview CodeAccessTier = "Review" + CodeAccessFullAccess CodeAccessTier = "FullAccess" ) +// SeatCode is an agent seat's access to the company's code. It maps onto +// Crewlet's role `code`, which holds one tier for every code host the engine +// drives: the seat's GitHub App is created with, and mints every token at, +// the permissions of its tier, and Crewlet's own code host, which this API +// does not offer, caps what the org chart grants the seat by the same tier. +// The tier is a setting of the seat's rather than of its GitHub identity, so +// it says the same on every host. A person acts as their own identity on +// every host, so a tier on a Human seat would cap nothing, and the API +// refuses one. +// +kubebuilder:validation:MinProperties=1 +type SeatCode struct { + // access is how much of a repository the seat may do. Unset is + // ReadOnly, the least the engine hands out. Crewlet's `access`. The + // engine warns that the tier grants nothing where nothing applies it: + // in a company with no GitHub Integration, or for a seat without + // spec.integrations.github. + // +optional + Access CodeAccessTier `json:"access,omitempty"` +} + // GitHubApp is the GitHub App a seat acts as. An app carries exactly one bot // identity, so every agent seat that works on GitHub has its own. type GitHubApp struct { @@ -286,14 +314,10 @@ type GitHubApp struct { // GitHubSeatIntegration is a seat's identity on GitHub. It maps onto // Crewlet's role integrations.github and needs the company's GitHub -// Integration. +// Integration. How much the identity may do is the seat's spec.code.access, +// which its app is created with and mints every token at. // +kubebuilder:validation:MinProperties=1 type GitHubSeatIntegration struct { - // tier is how much the seat may do in a repository. Unset is ReadOnly. - // Crewlet's `tier`. - // +optional - Tier GitHubAccessTier `json:"tier,omitempty"` - // repos are the repositories the seat works in, as owner/name. Unset is // every repository the app's installation covers. Crewlet's `repos`. // +optional @@ -438,7 +462,7 @@ type SeatContact struct { // SeatSpec is one seat of a company: an AI agent or a person. It renders to // one entry of Crewlet's roles, nested under its unit's. // +kubebuilder:validation:XValidation:rule="self.kind != 'Agent' || (!has(self.contact) && !has(self.availability))",message="contact and availability apply to Human seats only" -// +kubebuilder:validation:XValidation:rule="self.kind != 'Human' || !(has(self.llm) || has(self.tokenBudget) || has(self.workers) || has(self.learningEnabled) || has(self.mcp) || has(self.sandbox) || has(self.integrations) || has(self.behavioralGuidelines) || has(self.project) || has(self.space) || has(self.placement) || has(self.paused))",message="a Human seat is never run by the engine: llm, tokenBudget, workers, learningEnabled, mcp, sandbox, integrations, behavioralGuidelines, project, space, placement and paused apply to Agent seats only" +// +kubebuilder:validation:XValidation:rule="self.kind != 'Human' || !(has(self.llm) || has(self.tokenBudget) || has(self.workers) || has(self.learningEnabled) || has(self.mcp) || has(self.sandbox) || has(self.code) || has(self.integrations) || has(self.behavioralGuidelines) || has(self.project) || has(self.space) || has(self.placement) || has(self.paused))",message="a Human seat is never run by the engine: llm, tokenBudget, workers, learningEnabled, mcp, sandbox, code, integrations, behavioralGuidelines, project, space, placement and paused apply to Agent seats only" // +kubebuilder:validation:XValidation:rule="(!has(self.pauseReason) && !has(self.stopRunning)) || (has(self.paused) && self.paused)",message="pauseReason and stopRunning describe a pause: they apply only while paused is true" type SeatSpec struct { // companyRef names the Company this seat belongs to. Immutable. @@ -605,12 +629,23 @@ type SeatSpec struct { // stopRunning ends the turn the seat is on at its next round when it is // paused, rather than letting the turn finish; the stopped turn is not - // run again. Only with paused true. Setting it on a seat already paused - // stops the turn it is on then; unsetting it changes nothing, since a - // stopped turn stays stopped. Crewlet's pause `stop_running`. + // run again. A coding run the seat detached runs on, but every change + // it would make is refused until the seat is resumed: a call it makes + // through the tool bridge of any of the seat's tools not marked + // read-only, and a push its box makes to the company's code host, which + // the operator does not offer. Only with paused true. Setting it on a + // seat already paused stops the turn it is on then, and the writes of + // its runs; unsetting it changes nothing, since a stop lasts until the + // seat is resumed. Crewlet's pause `stop_running`. // +optional StopRunning *bool `json:"stopRunning,omitempty"` + // code is the seat's access to the company's code, on every code host + // the engine drives. Unset runs the seat at access ReadOnly. Crewlet's + // `code`. + // +optional + Code SeatCode `json:"code,omitzero"` + // integrations is the seat's own identity on the company's // integrations. // +optional @@ -737,7 +772,8 @@ type SeatPauseStatus struct { // +kubebuilder:validation:MaxLength=500 Reason string `json:"reason,omitempty"` - // stopRunning is whether the pause also ended the turn the seat was on. + // stopRunning is whether the pause also ended the turn the seat was on, + // and refuses its detached coding runs' writes until it is resumed. // +optional StopRunning *bool `json:"stopRunning,omitempty"` } diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 5d8b0c6..da822e9 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -344,6 +344,7 @@ func (in *CompanySpec) DeepCopyInto(out *CompanySpec) { } out.Tracker = in.Tracker in.Knowledge.DeepCopyInto(&out.Knowledge) + out.Inbox = in.Inbox out.TokenBudget = in.TokenBudget in.LLM.DeepCopyInto(&out.LLM) out.Embeddings = in.Embeddings @@ -1275,6 +1276,21 @@ func (in *GitLabSeatAccess) DeepCopy() *GitLabSeatAccess { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *InboxSettings) DeepCopyInto(out *InboxSettings) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new InboxSettings. +func (in *InboxSettings) DeepCopy() *InboxSettings { + if in == nil { + return nil + } + out := new(InboxSettings) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *IngressExposure) DeepCopyInto(out *IngressExposure) { *out = *in @@ -2210,6 +2226,22 @@ func (in *S3ObjectSettings) DeepCopy() *S3ObjectSettings { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *SandboxEnvVar) DeepCopyInto(out *SandboxEnvVar) { + *out = *in + out.ValueFrom = in.ValueFrom +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SandboxEnvVar. +func (in *SandboxEnvVar) DeepCopy() *SandboxEnvVar { + if in == nil { + return nil + } + out := new(SandboxEnvVar) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *SandboxFile) DeepCopyInto(out *SandboxFile) { *out = *in @@ -2274,7 +2306,7 @@ func (in *SandboxSetupStep) DeepCopyInto(out *SandboxSetupStep) { } if in.Env != nil { in, out := &in.Env, &out.Env - *out = make([]SecretEnvVar, len(*in)) + *out = make([]SandboxEnvVar, len(*in)) copy(*out, *in) } if in.OptionalKeys != nil { @@ -2446,6 +2478,21 @@ func (in *Seat) DeepCopyObject() runtime.Object { return nil } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *SeatCode) DeepCopyInto(out *SeatCode) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SeatCode. +func (in *SeatCode) DeepCopy() *SeatCode { + if in == nil { + return nil + } + out := new(SeatCode) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *SeatContact) DeepCopyInto(out *SeatContact) { *out = *in @@ -2863,6 +2910,7 @@ func (in *SeatSpec) DeepCopyInto(out *SeatSpec) { *out = new(bool) **out = **in } + out.Code = in.Code in.Integrations.DeepCopyInto(&out.Integrations) out.Contact = in.Contact } @@ -3138,7 +3186,6 @@ func (in *TopologyStatus) DeepCopy() *TopologyStatus { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *TrackerSettings) DeepCopyInto(out *TrackerSettings) { *out = *in - out.Worklet = in.Worklet } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new TrackerSettings. @@ -3298,18 +3345,3 @@ func (in *WebhookExposure) DeepCopy() *WebhookExposure { in.DeepCopyInto(out) return out } - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *WorkletSettings) DeepCopyInto(out *WorkletSettings) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new WorkletSettings. -func (in *WorkletSettings) DeepCopy() *WorkletSettings { - if in == nil { - return nil - } - out := new(WorkletSettings) - in.DeepCopyInto(out) - return out -} diff --git a/config/crd/bases/crewlet.ai_companies.yaml b/config/crd/bases/crewlet.ai_companies.yaml index ff9a18a..b2df960 100644 --- a/config/crd/bases/crewlet.ai_companies.yaml +++ b/config/crd/bases/crewlet.ai_companies.yaml @@ -246,12 +246,18 @@ spec: apply, and a refusal is reported on the Company. It may not set what another field or object of this API owns: name, - mission, vision, policies, timezone, tracker, knowledge, token_budget, - providers, integrations, mcp_servers, roles or units. Every position - Crewlet holds a credential in lives under one of those keys, so - engineConfig can never carry a credential: the API server refuses a - Company whose engineConfig names one of them, before anything is - stored. + mission, vision, policies, timezone, tracker, knowledge, inbox, + token_budget, providers, integrations, mcp_servers, roles or units. + Every position Crewlet holds a credential in lives under one of those + keys, so engineConfig can never carry a credential: the API server + refuses a Company whose engineConfig names one of them, before anything + is stored. + + Nor may it set code, Crewlet's code host, which keeps the company's git + repositories in the engine and which v1alpha1 does not offer: the host + serves them only from a node with the git role and a cache of them on + its volume, and no pod the operator runs has either, so a document + that set it would declare repositories nothing can clone or push. items: description: |- EngineConfigEntry is one top-level key of the company document that @@ -280,12 +286,17 @@ spec: x-kubernetes-list-type: map x-kubernetes-validations: - message: 'engineConfig may not set name, mission, vision, policies, - timezone, tracker, knowledge, token_budget, providers, integrations, - mcp_servers, roles or units: those are fields and objects of this - API, and the only blocks that hold credentials' + timezone, tracker, knowledge, inbox, token_budget, providers, + integrations, mcp_servers, roles or units: those are fields and + objects of this API, and the only blocks that hold credentials' rule: self.all(e, !(e.name in ['name', 'mission', 'vision', 'policies', - 'timezone', 'tracker', 'knowledge', 'token_budget', 'providers', + 'timezone', 'tracker', 'knowledge', 'inbox', 'token_budget', 'providers', 'integrations', 'mcp_servers', 'roles', 'units'])) + - message: 'engineConfig may not set code: Crewlet''s code host is + not offered in v1alpha1, since no pod the operator runs has the + git role or the cache that serves a company''s repositories, which + nothing could clone or push; remove the entry' + rule: self.all(e, e.name != 'code') exposure: description: exposure overrides how the company is reached. minProperties: 1 @@ -425,6 +436,17 @@ spec: is Go's net/url as the API server was built and is run: it takes such a zone, and under Go's GODEBUG setting urlstrictcolons=0 https://h.example:80:80 too. + + The engine also hands it to its sandboxes, as its tool bridge's address + (CREWLET_MCP_BRIDGE_URL) while an LLMProvider of the company runs in + mode Agent, and as its telemetry receiver's + (CREWLET_SANDBOX_OTEL_RECEIVER_URL) while the class sends telemetry to + a collector, and refuses to start on one the URL parser of a sandbox's + coding agent refuses, the WHATWG URL Standard's. So its host may not + hold an IPv6 zone or another '%', a '<', a '>' or a ']' outside an + IPv6 address's brackets, and a host whose last label is a number must + be an IPv4 address written as four decimal octets: that parser reads + such a host as an address, and Crewlet refuses one that is none. maxLength: 2048 minLength: 1 type: string @@ -443,6 +465,25 @@ spec: address, or starts on one no vendor reaches (write https://[2001:db8::1]:8443, not https://2001:db8::1:8443)' rule: '!self.matches(''^[^:/?#]+://'') || self.matches(''^[^:/?#]+://([^/?#]*@)?([[](([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|([0-9A-Fa-f]{1,4}:){1,7}:|:(:[0-9A-Fa-f]{1,4}){1,7}|::|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){1,6}|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){1,5}|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){1,4}|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){1,3}|([0-9A-Fa-f]{1,4}:){5}(:[0-9A-Fa-f]{1,4}){1,2}|([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){6}|:(:[0-9A-Fa-f]{1,4}){0,5}:|([0-9A-Fa-f]{1,4}:){5}:|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){0,4}:|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){0,3}:|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){0,2}:|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){0,1}:)(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])([.](25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])){3})(%25[^]/?#]+)?[]]|[^/?#:[]*)(:[0-9]*)?([/?#]|$)'')' + - message: 'publicBaseURL''s host may not hold a ''%'': the URL + parser of a sandbox''s coding agent refuses an IPv6 zone (%25eth0) + and a character no host may hold, and Crewlet refuses to start + when it hands a sandbox such an address; write the host as + it is, and an IPv6 address without a zone' + rule: '!isURL(self) || !self.matches(''^https?://[^/]*%'')' + - message: 'publicBaseURL''s host may not hold a ''<'', a ''>'' + or a '']'': the URL parser of a sandbox''s coding agent refuses + each in a host name, as it does an unfilled placeholder such + as https://, and Crewlet refuses to start when + it hands a sandbox such an address' + rule: '!isURL(self) || !self.matches(''^https?://[^/[]*[]<>]'')' + - message: 'publicBaseURL''s host ends in a number, which the + URL parser of a sandbox''s coding agent reads as an IPv4 address, + and Crewlet refuses to start when it hands a sandbox one that + is none: write an IPv4 address as four decimal octets, such + as 192.0.2.10, and a host name whose last label is not a number' + rule: '!isURL(self) || !self.matches(''^https?://([^/:]*[.])?([0-9]+|0[xX][0-9a-fA-F]*)[.]?(:[0-9]*)?(/|$)'') + || self.matches(''^https?://((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])[.]){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(:[0-9]*)?(/|$)'')' - message: 'publicBaseURL may not hold a user name or password: a credential is named through a Secret, never written into an address' @@ -458,6 +499,28 @@ spec: to name' rule: '!self.matches(''[$][{][A-Za-z_][A-Za-z0-9_]*[}]'')' type: object + inbox: + description: inbox is the company's policy over a person's inbox. + minProperties: 1 + properties: + retentionSeconds: + description: |- + retentionSeconds is how long a person's inbox keeps a notice; the + history the notice points at is kept regardless. Crewlet keeps the + horizon in whole days, so it is a whole number of days in seconds, + from 30 days (2592000) to 3650 days (315360000), such as 7776000 for + 90 days. Unset takes Crewlet's default of 365 days. Crewlet's + `inbox.retention_days`, which is this divided by 86400. + format: int32 + maximum: 315360000 + minimum: 2592000 + type: integer + x-kubernetes-validations: + - message: 'retentionSeconds must be whole days, a multiple of + 86400 such as 7776000 for 90 days: Crewlet keeps a person''s + inbox for whole days' + rule: self % 86400 == 0 + type: object integrations: description: integrations is company-wide integration behaviour. minProperties: 1 @@ -949,20 +1012,45 @@ spec: env: description: |- env is merged into the environment of the commands and of the run. - Crewlet's `env`, a credential position. + It may not set a variable Crewlet sets in a coding run itself, which + Crewlet refuses in every company document, as a Seat's + spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL, + CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and + CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's + `env`, a credential position. items: description: |- - SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + SandboxEnvVar is a variable a sandbox setup step merges into the + environment of its commands and of a coding run. Crewlet treats every value + there as a credential, so a literal value is not accepted: the value always + comes from a Secret. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding + run itself, as the run''s credential on the company''s + own code host, the base its clones and pushes + reach and the values its commit hook writes into + every commit''s trailers, and refuses a company + document that sets one, since a value here would + break the run''s clones and pushes or make a commit''s + trailers disagree with its push''s record; remove + it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is the Secret key the value is read from. @@ -1345,24 +1433,7 @@ spec: - Jira - None type: string - worklet: - description: worklet is the policy of Worklet, for backend Worklet. - minProperties: 1 - properties: - inboxRetentionDays: - description: |- - inboxRetentionDays is how long a person's inbox keeps a row; the - history the row points at is kept regardless. Unset takes Crewlet's - default of 365. Crewlet's `inbox_retention_days`. - format: int32 - maximum: 3650 - minimum: 30 - type: integer - type: object type: object - x-kubernetes-validations: - - message: worklet applies to backend Worklet only - rule: '!has(self.worklet) || self.backend == ''Worklet''' vision: description: |- vision is a longer statement of where the company is going, in diff --git a/config/crd/bases/crewlet.ai_engineclasses.yaml b/config/crd/bases/crewlet.ai_engineclasses.yaml index 51f1e9d..7eaf024 100644 --- a/config/crd/bases/crewlet.ai_engineclasses.yaml +++ b/config/crd/bases/crewlet.ai_engineclasses.yaml @@ -219,7 +219,7 @@ spec: that hung. It is at least every bound Crewlet puts on a data pod's boot, laid end - to end, which a window may not cut short: 1636 seconds, or 1776 with + to end, which a window may not cut short: 1461 seconds, or 1601 with objects.backend S3, whose bucket every boot probes. Those are its waits for its own broker and store (provisioning the event stream, the coordination buckets, the object store and the state logs, two minutes @@ -227,10 +227,13 @@ spec: around them), never for a peer or a seat pod. One of them grows with what the node holds and is left to the recovery's share, as the recovery is: the claim of each batch of a seat pod's events the node - kept and could not settle, 15 s a batch. Unset derives it from the - data volume: that floor plus six seconds per GiB of the volume. That is - four times the second and a half per GiB Crewlet measured recovering - after a crash on a fast local disk, for volumes that are + kept and could not settle, 15 s a batch. And a read of a state log's + state or record is counted at the 30 s Crewlet asks it for while the + broker answers none, past which it gives the read up, so no broker + that never answers holds a boot past the window. Unset derives it from + the data volume: that floor plus six seconds per GiB of the volume. + That is four times the second and a half per GiB Crewlet measured + recovering after a crash on a fast local disk, for volumes that are network-attached; set this for disks slower than that. Nothing the company configures counts: the engine answers /health once it has recovered, and only then reads the company's secrets, starts its tool @@ -238,7 +241,7 @@ spec: Changing it changes the data pods' template, which restarts them. format: int32 maximum: 86400 - minimum: 1636 + minimum: 1461 type: integer storage: description: storage is the data volume each data pod gets. @@ -256,16 +259,18 @@ spec: stream.store_max_bytes: the stream reserves what a node keeps there up front, and a fresh node refuses to boot on a share that cannot hold it (Crewlet's engine.BrokerFloor, and its deployment guide's "minimum room - for a fresh node"). Its seven state logs' ceilings at their smallest - come to 6979321856 bytes, so a volume must be at least 9Gi, whose share - of 6.75Gi holds them. With objects.backend NATS, the default, the object - store's bucket reserves a quarter of the share beside them, 9305762474 - bytes in all, so such a class must give at least 12Gi, whose share is - 9Gi. That is the least that boots: the mailboxes and coordination - buckets grow into what is left, so give a company that runs for long - more. The stream's ceiling is 64Ti (config.StoreMaxBytesCeiling, a - guard against a unit mistake), so a volume larger than about 85Ti gives - the stream 64Ti and the rest to the store and its snapshots. + for a fresh node"). Its eight state logs' ceilings at their smallest + (the code host's log among them, which a node reserves whether or not + the company hosts code) come to 8053063680 bytes, so a volume must be + at least 10Gi, whose share of 7.5Gi holds them. With objects.backend + NATS, the default, the object store's bucket reserves a quarter of the + share beside them, 10737418239 bytes in all, so such a class must give + at least 14Gi, whose share is 10.5Gi. That is the least that boots: the + mailboxes and coordination buckets grow into what is left, so give a + company that runs for long more. The stream's ceiling is 64Ti + (config.StoreMaxBytesCeiling, a guard against a unit mistake), so a + volume larger than about 85Ti gives the stream 64Ti and the rest to the + store and its snapshots. It applies to the data volumes of companies created after it is set. An existing company's volumes keep the size they were created with: @@ -274,14 +279,20 @@ spec: decision about one company's storage class and data, not the class's. Its owner expands the company's claim instead, and once the expansion lands the operator sizes the stream from what the claim then has. + + A class stored below the floor, under an older one, keeps its size, + and every Company created under it gets a volume no fresh node boots + on, so raise it. Until it is raised the API refuses any change to the + class's spec, or, with objects.backend S3, a change of its size that + leaves it below the floor. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - - message: 'size must be at least 9Gi: three quarters of the - volume is the engine''s event stream, and a fresh node - of Crewlet reserves at least 6979321856 bytes of it for - its state logs' - rule: '!quantity(string(self)).isLessThan(quantity(''9Gi''))' + - message: 'size must be at least 10Gi: three quarters of + the volume is the engine''s event stream, and a fresh + node of Crewlet reserves at least 8053063680 bytes of + it for its state logs' + rule: '!quantity(string(self)).isLessThan(quantity(''10Gi''))' storageClassName: description: |- storageClassName is the StorageClass of the data volumes. Unset uses @@ -458,7 +469,7 @@ spec: refuses to start on a browser's address whose host ends in a label that is a number: a browser reads such a host as an IPv4 address and rewrites it before it sends it. So its last label may not be one - ({{uid}}.123, {{uid}}.0x1f), where webhooks.hostname's may. A last + ({{uid}}.123, {{uid}}.0x1f), as webhooks.hostname's may not. A last label a variable fills is a number only for some Companies: one whose namespace is a number (4711, 0x1f) under {{name}}.{{namespace}}, or whose namespace or name makes one with the template's own characters @@ -1010,6 +1021,15 @@ spec: {{name}} and {{namespace}} joined by a '.', which neither can hold, so that no two companies share a host name: with a '-' between them, team-a/b-web and team-a-b/web would. + + The engine hands the address to its sandboxes too, as its tool + bridge's and its telemetry receiver's, and refuses to start on one the + URL parser of a sandbox's coding agent reads as an IPv4 address that + is none, as it does a host whose last label is a number. So that last + label may not be one ({{uid}}.123, {{uid}}.0x1f). One a variable fills + is a number only for some Companies, as api.hostname says, and the + operator holds that Company's engine back, with RuntimeReady=False + AddressRefused, while it hands the address to a sandbox. maxLength: 253 minLength: 1 type: string @@ -1029,6 +1049,13 @@ spec: variables are replaced rule: self.replace('{{namespace}}', 'n').replace('{{name}}', 'n').replace('{{uid}}', 'u').matches('^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$') + - message: 'hostname may not end in a label that is a number, + such as {{uid}}.123 or {{uid}}.0x1f: a sandbox''s URL + parser reads the host as an IPv4 address, and Crewlet + refuses to start when it hands a sandbox one that is none; + end it in a label that is not one, such as a domain of + your own' + rule: '!self.matches(''(^|[.])([0-9]+|0x[0-9a-f]*)$'')' ingress: description: ingress is the Ingress settings, for mode Ingress. properties: @@ -3530,18 +3557,18 @@ spec: - engine type: object x-kubernetes-validations: - - message: 'data.storage.size must be at least 12Gi with objects.backend + - message: 'data.storage.size must be at least 14Gi with objects.backend NATS: three quarters of the volume is the engine''s event stream, - and a fresh node of Crewlet reserves at least 9305762474 bytes of + and a fresh node of Crewlet reserves at least 10737418239 bytes of it for its state logs and the object store''s bucket' rule: (has(self.objects) && has(self.objects.backend) && self.objects.backend - == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('12Gi')) - - message: 'data.startupSeconds must be at least 1776 with objects.backend + == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('14Gi')) + - message: 'data.startupSeconds must be at least 1601 with objects.backend S3: every boot of a data pod probes the bucket, and Crewlet''s bounds - on the boot then add up to 1776 seconds, which a shorter window can + on the boot then add up to 1601 seconds, which a shorter window can cut short' rule: '!has(self.data.startupSeconds) || !has(self.objects) || !has(self.objects.backend) - || self.objects.backend != ''S3'' || self.data.startupSeconds >= 1776' + || self.objects.backend != ''S3'' || self.data.startupSeconds >= 1601' - message: 'networkPolicy Isolated with an Ingress needs ingress.controllerNamespaces: without them the policy refuses everything the Ingress forwards' rule: self.networkPolicy.mode != 'Isolated' || ((!has(self.exposure.webhooks.ingress) diff --git a/config/crd/bases/crewlet.ai_llmproviders.yaml b/config/crd/bases/crewlet.ai_llmproviders.yaml index d68c045..0b00db8 100644 --- a/config/crd/bases/crewlet.ai_llmproviders.yaml +++ b/config/crd/bases/crewlet.ai_llmproviders.yaml @@ -232,9 +232,9 @@ spec: items: description: |- SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + such as a coding CLI's model key. Crewlet treats every value in the + positions that use this type as a credential, so a literal value is not + accepted: the value always comes from a Secret. properties: name: description: name is the variable's name. diff --git a/config/crd/bases/crewlet.ai_seats.yaml b/config/crd/bases/crewlet.ai_seats.yaml index 290c2ad..aec16a9 100644 --- a/config/crd/bases/crewlet.ai_seats.yaml +++ b/config/crd/bases/crewlet.ai_seats.yaml @@ -111,6 +111,26 @@ spec: maxItems: 32 type: array x-kubernetes-list-type: atomic + code: + description: |- + code is the seat's access to the company's code, on every code host + the engine drives. Unset runs the seat at access ReadOnly. Crewlet's + `code`. + minProperties: 1 + properties: + access: + description: |- + access is how much of a repository the seat may do. Unset is + ReadOnly, the least the engine hands out. Crewlet's `access`. The + engine warns that the tier grants nothing where nothing applies it: + in a company with no GitHub Integration, or for a seat without + spec.integrations.github. + enum: + - ReadOnly + - Review + - FullAccess + type: string + type: object companyRef: description: companyRef names the Company this seat belongs to. Immutable. properties: @@ -316,15 +336,6 @@ spec: maxItems: 64 type: array x-kubernetes-list-type: set - tier: - description: |- - tier is how much the seat may do in a repository. Unset is ReadOnly. - Crewlet's `tier`. - enum: - - ReadOnly - - Review - - FullAccess - type: string type: object mattermost: description: mattermost is the seat's own Mattermost bot. @@ -955,19 +966,47 @@ spec: endpoint of the provider the code work runs on out of the box, unless a run goes without its value (optionalKeys), when the box is given them as it is with no sign-in; and a variable named as one the - provider would set wins over it. Crewlet's `env`, a credential - position. + provider would set wins over it. It may not set a variable Crewlet + sets in a coding run itself, which Crewlet refuses in every company + document, and nor may a setup step's env: CREWLET_GIT_TOKEN and + CREWLET_GIT_URL, a run's credential on the company's own code host and + the base its clones and pushes reach, and CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM, the values + its commit hook writes into every commit's trailers (Crewlet's + ReservedSandboxEnv). No run of a company the operator runs is given + them, since the operator does not offer the code host, but the names + are the engine's all the same. Crewlet's `env`, a credential position. items: description: |- SeatEnvVar is a named credential of a seat delivered as an environment - variable: from a Secret, or provisioned by the engine. + variable of its sandbox runs (a Seat's spec.sandbox.env): from a Secret, or + provisioned by the engine. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding run + itself, as the run''s credential on the company''s own + code host, the base its clones and pushes reach and + the values its commit hook writes into every commit''s + trailers, and refuses a company document that sets one, + since a value here would break the run''s clones and + pushes or make a commit''s trailers disagree with its + push''s record; remove it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is where the value comes from. minProperties: 1 @@ -1146,20 +1185,45 @@ spec: env: description: |- env is merged into the environment of the commands and of the run. - Crewlet's `env`, a credential position. + It may not set a variable Crewlet sets in a coding run itself, which + Crewlet refuses in every company document, as a Seat's + spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL, + CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and + CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's + `env`, a credential position. items: description: |- - SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + SandboxEnvVar is a variable a sandbox setup step merges into the + environment of its commands and of a coding run. Crewlet treats every value + there as a credential, so a literal value is not accepted: the value always + comes from a Secret. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding + run itself, as the run''s credential on the company''s + own code host, the base its clones and pushes + reach and the values its commit hook writes into + every commit''s trailers, and refuses a company + document that sets one, since a value here would + break the run''s clones and pushes or make a commit''s + trailers disagree with its push''s record; remove + it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is the Secret key the value is read from. @@ -1372,9 +1436,14 @@ spec: description: |- stopRunning ends the turn the seat is on at its next round when it is paused, rather than letting the turn finish; the stopped turn is not - run again. Only with paused true. Setting it on a seat already paused - stops the turn it is on then; unsetting it changes nothing, since a - stopped turn stays stopped. Crewlet's pause `stop_running`. + run again. A coding run the seat detached runs on, but every change + it would make is refused until the seat is resumed: a call it makes + through the tool bridge of any of the seat's tools not marked + read-only, and a push its box makes to the company's code host, which + the operator does not offer. Only with paused true. Setting it on a + seat already paused stops the turn it is on then, and the writes of + its runs; unsetting it changes nothing, since a stop lasts until the + seat is resumed. Crewlet's pause `stop_running`. type: boolean tokenBudget: description: |- @@ -1434,13 +1503,13 @@ spec: - message: contact and availability apply to Human seats only rule: self.kind != 'Agent' || (!has(self.contact) && !has(self.availability)) - message: 'a Human seat is never run by the engine: llm, tokenBudget, - workers, learningEnabled, mcp, sandbox, integrations, behavioralGuidelines, + workers, learningEnabled, mcp, sandbox, code, integrations, behavioralGuidelines, project, space, placement and paused apply to Agent seats only' rule: self.kind != 'Human' || !(has(self.llm) || has(self.tokenBudget) || has(self.workers) || has(self.learningEnabled) || has(self.mcp) - || has(self.sandbox) || has(self.integrations) || has(self.behavioralGuidelines) - || has(self.project) || has(self.space) || has(self.placement) || - has(self.paused)) + || has(self.sandbox) || has(self.code) || has(self.integrations) || + has(self.behavioralGuidelines) || has(self.project) || has(self.space) + || has(self.placement) || has(self.paused)) - message: 'pauseReason and stopRunning describe a pause: they apply only while paused is true' rule: (!has(self.pauseReason) && !has(self.stopRunning)) || (has(self.paused) @@ -1601,8 +1670,9 @@ spec: minLength: 1 type: string stopRunning: - description: stopRunning is whether the pause also ended the turn - the seat was on. + description: |- + stopRunning is whether the pause also ended the turn the seat was on, + and refuses its detached coding runs' writes until it is resumed. type: boolean type: object placement: diff --git a/config/samples/v1alpha1_seat.yaml b/config/samples/v1alpha1_seat.yaml index ff2e322..9f7ca60 100644 --- a/config/samples/v1alpha1_seat.yaml +++ b/config/samples/v1alpha1_seat.yaml @@ -103,13 +103,16 @@ spec: valueFrom: provisioned: name: gitlab-token + # How much of a repository the seat may do, on GitHub through its app + # below. + code: + access: Review integrations: slack: # Keys: botToken, signingSecret. credentialsSecretRef: name: ana-slack github: - tier: Review repos: - nimbus-hq/nimbuscore app: diff --git a/dist/chart/templates/crd/companies.crewlet.ai.yaml b/dist/chart/templates/crd/companies.crewlet.ai.yaml index 3cb434b..1eebe6c 100644 --- a/dist/chart/templates/crd/companies.crewlet.ai.yaml +++ b/dist/chart/templates/crd/companies.crewlet.ai.yaml @@ -249,12 +249,18 @@ spec: apply, and a refusal is reported on the Company. It may not set what another field or object of this API owns: name, - mission, vision, policies, timezone, tracker, knowledge, token_budget, - providers, integrations, mcp_servers, roles or units. Every position - Crewlet holds a credential in lives under one of those keys, so - engineConfig can never carry a credential: the API server refuses a - Company whose engineConfig names one of them, before anything is - stored. + mission, vision, policies, timezone, tracker, knowledge, inbox, + token_budget, providers, integrations, mcp_servers, roles or units. + Every position Crewlet holds a credential in lives under one of those + keys, so engineConfig can never carry a credential: the API server + refuses a Company whose engineConfig names one of them, before anything + is stored. + + Nor may it set code, Crewlet's code host, which keeps the company's git + repositories in the engine and which v1alpha1 does not offer: the host + serves them only from a node with the git role and a cache of them on + its volume, and no pod the operator runs has either, so a document + that set it would declare repositories nothing can clone or push. items: description: |- EngineConfigEntry is one top-level key of the company document that @@ -283,12 +289,17 @@ spec: x-kubernetes-list-type: map x-kubernetes-validations: - message: 'engineConfig may not set name, mission, vision, policies, - timezone, tracker, knowledge, token_budget, providers, integrations, - mcp_servers, roles or units: those are fields and objects of this - API, and the only blocks that hold credentials' + timezone, tracker, knowledge, inbox, token_budget, providers, + integrations, mcp_servers, roles or units: those are fields and + objects of this API, and the only blocks that hold credentials' rule: self.all(e, !(e.name in ['name', 'mission', 'vision', 'policies', - 'timezone', 'tracker', 'knowledge', 'token_budget', 'providers', + 'timezone', 'tracker', 'knowledge', 'inbox', 'token_budget', 'providers', 'integrations', 'mcp_servers', 'roles', 'units'])) + - message: 'engineConfig may not set code: Crewlet''s code host is + not offered in v1alpha1, since no pod the operator runs has the + git role or the cache that serves a company''s repositories, which + nothing could clone or push; remove the entry' + rule: self.all(e, e.name != 'code') exposure: description: exposure overrides how the company is reached. minProperties: 1 @@ -428,6 +439,17 @@ spec: is Go's net/url as the API server was built and is run: it takes such a zone, and under Go's GODEBUG setting urlstrictcolons=0 https://h.example:80:80 too. + + The engine also hands it to its sandboxes, as its tool bridge's address + (CREWLET_MCP_BRIDGE_URL) while an LLMProvider of the company runs in + mode Agent, and as its telemetry receiver's + (CREWLET_SANDBOX_OTEL_RECEIVER_URL) while the class sends telemetry to + a collector, and refuses to start on one the URL parser of a sandbox's + coding agent refuses, the WHATWG URL Standard's. So its host may not + hold an IPv6 zone or another '%', a '<', a '>' or a ']' outside an + IPv6 address's brackets, and a host whose last label is a number must + be an IPv4 address written as four decimal octets: that parser reads + such a host as an address, and Crewlet refuses one that is none. maxLength: 2048 minLength: 1 type: string @@ -446,6 +468,25 @@ spec: address, or starts on one no vendor reaches (write https://[2001:db8::1]:8443, not https://2001:db8::1:8443)' rule: '!self.matches(''^[^:/?#]+://'') || self.matches(''^[^:/?#]+://([^/?#]*@)?([[](([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|([0-9A-Fa-f]{1,4}:){1,7}:|:(:[0-9A-Fa-f]{1,4}){1,7}|::|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){1,6}|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){1,5}|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){1,4}|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){1,3}|([0-9A-Fa-f]{1,4}:){5}(:[0-9A-Fa-f]{1,4}){1,2}|([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4})|(([0-9A-Fa-f]{1,4}:){6}|:(:[0-9A-Fa-f]{1,4}){0,5}:|([0-9A-Fa-f]{1,4}:){5}:|([0-9A-Fa-f]{1,4}:)(:[0-9A-Fa-f]{1,4}){0,4}:|([0-9A-Fa-f]{1,4}:){2}(:[0-9A-Fa-f]{1,4}){0,3}:|([0-9A-Fa-f]{1,4}:){3}(:[0-9A-Fa-f]{1,4}){0,2}:|([0-9A-Fa-f]{1,4}:){4}(:[0-9A-Fa-f]{1,4}){0,1}:)(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])([.](25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])){3})(%25[^]/?#]+)?[]]|[^/?#:[]*)(:[0-9]*)?([/?#]|$)'')' + - message: 'publicBaseURL''s host may not hold a ''%'': the URL + parser of a sandbox''s coding agent refuses an IPv6 zone (%25eth0) + and a character no host may hold, and Crewlet refuses to start + when it hands a sandbox such an address; write the host as + it is, and an IPv6 address without a zone' + rule: '!isURL(self) || !self.matches(''^https?://[^/]*%'')' + - message: 'publicBaseURL''s host may not hold a ''<'', a ''>'' + or a '']'': the URL parser of a sandbox''s coding agent refuses + each in a host name, as it does an unfilled placeholder such + as https://, and Crewlet refuses to start when + it hands a sandbox such an address' + rule: '!isURL(self) || !self.matches(''^https?://[^/[]*[]<>]'')' + - message: 'publicBaseURL''s host ends in a number, which the + URL parser of a sandbox''s coding agent reads as an IPv4 address, + and Crewlet refuses to start when it hands a sandbox one that + is none: write an IPv4 address as four decimal octets, such + as 192.0.2.10, and a host name whose last label is not a number' + rule: '!isURL(self) || !self.matches(''^https?://([^/:]*[.])?([0-9]+|0[xX][0-9a-fA-F]*)[.]?(:[0-9]*)?(/|$)'') + || self.matches(''^https?://((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])[.]){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(:[0-9]*)?(/|$)'')' - message: 'publicBaseURL may not hold a user name or password: a credential is named through a Secret, never written into an address' @@ -461,6 +502,28 @@ spec: to name' rule: '!self.matches(''[$][{][A-Za-z_][A-Za-z0-9_]*[}]'')' type: object + inbox: + description: inbox is the company's policy over a person's inbox. + minProperties: 1 + properties: + retentionSeconds: + description: |- + retentionSeconds is how long a person's inbox keeps a notice; the + history the notice points at is kept regardless. Crewlet keeps the + horizon in whole days, so it is a whole number of days in seconds, + from 30 days (2592000) to 3650 days (315360000), such as 7776000 for + 90 days. Unset takes Crewlet's default of 365 days. Crewlet's + `inbox.retention_days`, which is this divided by 86400. + format: int32 + maximum: 315360000 + minimum: 2592000 + type: integer + x-kubernetes-validations: + - message: 'retentionSeconds must be whole days, a multiple of + 86400 such as 7776000 for 90 days: Crewlet keeps a person''s + inbox for whole days' + rule: self % 86400 == 0 + type: object integrations: description: integrations is company-wide integration behaviour. minProperties: 1 @@ -952,20 +1015,45 @@ spec: env: description: |- env is merged into the environment of the commands and of the run. - Crewlet's `env`, a credential position. + It may not set a variable Crewlet sets in a coding run itself, which + Crewlet refuses in every company document, as a Seat's + spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL, + CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and + CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's + `env`, a credential position. items: description: |- - SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + SandboxEnvVar is a variable a sandbox setup step merges into the + environment of its commands and of a coding run. Crewlet treats every value + there as a credential, so a literal value is not accepted: the value always + comes from a Secret. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding + run itself, as the run''s credential on the company''s + own code host, the base its clones and pushes + reach and the values its commit hook writes into + every commit''s trailers, and refuses a company + document that sets one, since a value here would + break the run''s clones and pushes or make a commit''s + trailers disagree with its push''s record; remove + it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is the Secret key the value is read from. @@ -1348,24 +1436,7 @@ spec: - Jira - None type: string - worklet: - description: worklet is the policy of Worklet, for backend Worklet. - minProperties: 1 - properties: - inboxRetentionDays: - description: |- - inboxRetentionDays is how long a person's inbox keeps a row; the - history the row points at is kept regardless. Unset takes Crewlet's - default of 365. Crewlet's `inbox_retention_days`. - format: int32 - maximum: 3650 - minimum: 30 - type: integer - type: object type: object - x-kubernetes-validations: - - message: worklet applies to backend Worklet only - rule: '!has(self.worklet) || self.backend == ''Worklet''' vision: description: |- vision is a longer statement of where the company is going, in diff --git a/dist/chart/templates/crd/engineclasses.crewlet.ai.yaml b/dist/chart/templates/crd/engineclasses.crewlet.ai.yaml index 7b0d9b4..1427a78 100644 --- a/dist/chart/templates/crd/engineclasses.crewlet.ai.yaml +++ b/dist/chart/templates/crd/engineclasses.crewlet.ai.yaml @@ -222,7 +222,7 @@ spec: that hung. It is at least every bound Crewlet puts on a data pod's boot, laid end - to end, which a window may not cut short: 1636 seconds, or 1776 with + to end, which a window may not cut short: 1461 seconds, or 1601 with objects.backend S3, whose bucket every boot probes. Those are its waits for its own broker and store (provisioning the event stream, the coordination buckets, the object store and the state logs, two minutes @@ -230,10 +230,13 @@ spec: around them), never for a peer or a seat pod. One of them grows with what the node holds and is left to the recovery's share, as the recovery is: the claim of each batch of a seat pod's events the node - kept and could not settle, 15 s a batch. Unset derives it from the - data volume: that floor plus six seconds per GiB of the volume. That is - four times the second and a half per GiB Crewlet measured recovering - after a crash on a fast local disk, for volumes that are + kept and could not settle, 15 s a batch. And a read of a state log's + state or record is counted at the 30 s Crewlet asks it for while the + broker answers none, past which it gives the read up, so no broker + that never answers holds a boot past the window. Unset derives it from + the data volume: that floor plus six seconds per GiB of the volume. + That is four times the second and a half per GiB Crewlet measured + recovering after a crash on a fast local disk, for volumes that are network-attached; set this for disks slower than that. Nothing the company configures counts: the engine answers /health once it has recovered, and only then reads the company's secrets, starts its tool @@ -241,7 +244,7 @@ spec: Changing it changes the data pods' template, which restarts them. format: int32 maximum: 86400 - minimum: 1636 + minimum: 1461 type: integer storage: description: storage is the data volume each data pod gets. @@ -259,16 +262,18 @@ spec: stream.store_max_bytes: the stream reserves what a node keeps there up front, and a fresh node refuses to boot on a share that cannot hold it (Crewlet's engine.BrokerFloor, and its deployment guide's "minimum room - for a fresh node"). Its seven state logs' ceilings at their smallest - come to 6979321856 bytes, so a volume must be at least 9Gi, whose share - of 6.75Gi holds them. With objects.backend NATS, the default, the object - store's bucket reserves a quarter of the share beside them, 9305762474 - bytes in all, so such a class must give at least 12Gi, whose share is - 9Gi. That is the least that boots: the mailboxes and coordination - buckets grow into what is left, so give a company that runs for long - more. The stream's ceiling is 64Ti (config.StoreMaxBytesCeiling, a - guard against a unit mistake), so a volume larger than about 85Ti gives - the stream 64Ti and the rest to the store and its snapshots. + for a fresh node"). Its eight state logs' ceilings at their smallest + (the code host's log among them, which a node reserves whether or not + the company hosts code) come to 8053063680 bytes, so a volume must be + at least 10Gi, whose share of 7.5Gi holds them. With objects.backend + NATS, the default, the object store's bucket reserves a quarter of the + share beside them, 10737418239 bytes in all, so such a class must give + at least 14Gi, whose share is 10.5Gi. That is the least that boots: the + mailboxes and coordination buckets grow into what is left, so give a + company that runs for long more. The stream's ceiling is 64Ti + (config.StoreMaxBytesCeiling, a guard against a unit mistake), so a + volume larger than about 85Ti gives the stream 64Ti and the rest to the + store and its snapshots. It applies to the data volumes of companies created after it is set. An existing company's volumes keep the size they were created with: @@ -277,14 +282,20 @@ spec: decision about one company's storage class and data, not the class's. Its owner expands the company's claim instead, and once the expansion lands the operator sizes the stream from what the claim then has. + + A class stored below the floor, under an older one, keeps its size, + and every Company created under it gets a volume no fresh node boots + on, so raise it. Until it is raised the API refuses any change to the + class's spec, or, with objects.backend S3, a change of its size that + leaves it below the floor. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true x-kubernetes-validations: - - message: 'size must be at least 9Gi: three quarters of the - volume is the engine''s event stream, and a fresh node - of Crewlet reserves at least 6979321856 bytes of it for - its state logs' - rule: '!quantity(string(self)).isLessThan(quantity(''9Gi''))' + - message: 'size must be at least 10Gi: three quarters of + the volume is the engine''s event stream, and a fresh + node of Crewlet reserves at least 8053063680 bytes of + it for its state logs' + rule: '!quantity(string(self)).isLessThan(quantity(''10Gi''))' storageClassName: description: |- storageClassName is the StorageClass of the data volumes. Unset uses @@ -461,7 +472,7 @@ spec: refuses to start on a browser's address whose host ends in a label that is a number: a browser reads such a host as an IPv4 address and rewrites it before it sends it. So its last label may not be one - ({{ "{{uid}}" }}.123, {{ "{{uid}}" }}.0x1f), where webhooks.hostname's may. A last + ({{ "{{uid}}" }}.123, {{ "{{uid}}" }}.0x1f), as webhooks.hostname's may not. A last label a variable fills is a number only for some Companies: one whose namespace is a number (4711, 0x1f) under {{ "{{name}}" }}.{{ "{{namespace}}" }}, or whose namespace or name makes one with the template's own characters @@ -1013,6 +1024,15 @@ spec: {{ "{{name}}" }} and {{ "{{namespace}}" }} joined by a '.', which neither can hold, so that no two companies share a host name: with a '-' between them, team-a/b-web and team-a-b/web would. + + The engine hands the address to its sandboxes too, as its tool + bridge's and its telemetry receiver's, and refuses to start on one the + URL parser of a sandbox's coding agent reads as an IPv4 address that + is none, as it does a host whose last label is a number. So that last + label may not be one ({{ "{{uid}}" }}.123, {{ "{{uid}}" }}.0x1f). One a variable fills + is a number only for some Companies, as api.hostname says, and the + operator holds that Company's engine back, with RuntimeReady=False + AddressRefused, while it hands the address to a sandbox. maxLength: 253 minLength: 1 type: string @@ -1032,6 +1052,13 @@ spec: variables are replaced rule: self.replace('{{ "{{namespace}}" }}', 'n').replace('{{ "{{name}}" }}', 'n').replace('{{ "{{uid}}" }}', 'u').matches('^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$') + - message: 'hostname may not end in a label that is a number, + such as {{ "{{uid}}" }}.123 or {{ "{{uid}}" }}.0x1f: a sandbox''s URL + parser reads the host as an IPv4 address, and Crewlet + refuses to start when it hands a sandbox one that is none; + end it in a label that is not one, such as a domain of + your own' + rule: '!self.matches(''(^|[.])([0-9]+|0x[0-9a-f]*)$'')' ingress: description: ingress is the Ingress settings, for mode Ingress. properties: @@ -3533,18 +3560,18 @@ spec: - engine type: object x-kubernetes-validations: - - message: 'data.storage.size must be at least 12Gi with objects.backend + - message: 'data.storage.size must be at least 14Gi with objects.backend NATS: three quarters of the volume is the engine''s event stream, - and a fresh node of Crewlet reserves at least 9305762474 bytes of + and a fresh node of Crewlet reserves at least 10737418239 bytes of it for its state logs and the object store''s bucket' rule: (has(self.objects) && has(self.objects.backend) && self.objects.backend - == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('12Gi')) - - message: 'data.startupSeconds must be at least 1776 with objects.backend + == 'S3') || !has(self.data.storage.size) || !quantity(string(self.data.storage.size)).isLessThan(quantity('14Gi')) + - message: 'data.startupSeconds must be at least 1601 with objects.backend S3: every boot of a data pod probes the bucket, and Crewlet''s bounds - on the boot then add up to 1776 seconds, which a shorter window can + on the boot then add up to 1601 seconds, which a shorter window can cut short' rule: '!has(self.data.startupSeconds) || !has(self.objects) || !has(self.objects.backend) - || self.objects.backend != ''S3'' || self.data.startupSeconds >= 1776' + || self.objects.backend != ''S3'' || self.data.startupSeconds >= 1601' - message: 'networkPolicy Isolated with an Ingress needs ingress.controllerNamespaces: without them the policy refuses everything the Ingress forwards' rule: self.networkPolicy.mode != 'Isolated' || ((!has(self.exposure.webhooks.ingress) diff --git a/dist/chart/templates/crd/llmproviders.crewlet.ai.yaml b/dist/chart/templates/crd/llmproviders.crewlet.ai.yaml index 5567406..4b35540 100644 --- a/dist/chart/templates/crd/llmproviders.crewlet.ai.yaml +++ b/dist/chart/templates/crd/llmproviders.crewlet.ai.yaml @@ -235,9 +235,9 @@ spec: items: description: |- SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + such as a coding CLI's model key. Crewlet treats every value in the + positions that use this type as a credential, so a literal value is not + accepted: the value always comes from a Secret. properties: name: description: name is the variable's name. diff --git a/dist/chart/templates/crd/seats.crewlet.ai.yaml b/dist/chart/templates/crd/seats.crewlet.ai.yaml index cb390d8..16fead8 100644 --- a/dist/chart/templates/crd/seats.crewlet.ai.yaml +++ b/dist/chart/templates/crd/seats.crewlet.ai.yaml @@ -114,6 +114,26 @@ spec: maxItems: 32 type: array x-kubernetes-list-type: atomic + code: + description: |- + code is the seat's access to the company's code, on every code host + the engine drives. Unset runs the seat at access ReadOnly. Crewlet's + `code`. + minProperties: 1 + properties: + access: + description: |- + access is how much of a repository the seat may do. Unset is + ReadOnly, the least the engine hands out. Crewlet's `access`. The + engine warns that the tier grants nothing where nothing applies it: + in a company with no GitHub Integration, or for a seat without + spec.integrations.github. + enum: + - ReadOnly + - Review + - FullAccess + type: string + type: object companyRef: description: companyRef names the Company this seat belongs to. Immutable. properties: @@ -319,15 +339,6 @@ spec: maxItems: 64 type: array x-kubernetes-list-type: set - tier: - description: |- - tier is how much the seat may do in a repository. Unset is ReadOnly. - Crewlet's `tier`. - enum: - - ReadOnly - - Review - - FullAccess - type: string type: object mattermost: description: mattermost is the seat's own Mattermost bot. @@ -958,19 +969,47 @@ spec: endpoint of the provider the code work runs on out of the box, unless a run goes without its value (optionalKeys), when the box is given them as it is with no sign-in; and a variable named as one the - provider would set wins over it. Crewlet's `env`, a credential - position. + provider would set wins over it. It may not set a variable Crewlet + sets in a coding run itself, which Crewlet refuses in every company + document, and nor may a setup step's env: CREWLET_GIT_TOKEN and + CREWLET_GIT_URL, a run's credential on the company's own code host and + the base its clones and pushes reach, and CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM, the values + its commit hook writes into every commit's trailers (Crewlet's + ReservedSandboxEnv). No run of a company the operator runs is given + them, since the operator does not offer the code host, but the names + are the engine's all the same. Crewlet's `env`, a credential position. items: description: |- SeatEnvVar is a named credential of a seat delivered as an environment - variable: from a Secret, or provisioned by the engine. + variable of its sandbox runs (a Seat's spec.sandbox.env): from a Secret, or + provisioned by the engine. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding run + itself, as the run''s credential on the company''s own + code host, the base its clones and pushes reach and + the values its commit hook writes into every commit''s + trailers, and refuses a company document that sets one, + since a value here would break the run''s clones and + pushes or make a commit''s trailers disagree with its + push''s record; remove it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is where the value comes from. minProperties: 1 @@ -1149,20 +1188,45 @@ spec: env: description: |- env is merged into the environment of the commands and of the run. - Crewlet's `env`, a credential position. + It may not set a variable Crewlet sets in a coding run itself, which + Crewlet refuses in every company document, as a Seat's + spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL, + CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and + CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's + `env`, a credential position. items: description: |- - SecretEnvVar is a named credential delivered as an environment variable, - such as a sandbox setup step's token or a coding CLI's model key. Crewlet - treats every value in the positions that use this type as a credential, so - a literal value is not accepted: the value always comes from a Secret. + SandboxEnvVar is a variable a sandbox setup step merges into the + environment of its commands and of a coding run. Crewlet treats every value + there as a credential, so a literal value is not accepted: the value always + comes from a Secret. properties: name: - description: name is the variable's name. + description: |- + name is the variable's name. It may not be one Crewlet sets in a coding + run itself, which Crewlet refuses in every company document: + CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's + ReservedSandboxEnv). maxLength: 128 minLength: 1 pattern: ^[A-Za-z_][A-Za-z0-9_]*$ type: string + x-kubernetes-validations: + - message: 'CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT, + CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM + are the engine''s: Crewlet sets them in a coding + run itself, as the run''s credential on the company''s + own code host, the base its clones and pushes + reach and the values its commit hook writes into + every commit''s trailers, and refuses a company + document that sets one, since a value here would + break the run''s clones and pushes or make a commit''s + trailers disagree with its push''s record; remove + it' + rule: '!(self in [''CREWLET_GIT_TOKEN'', ''CREWLET_GIT_URL'', + ''CREWLET_GIT_TRAILER_SEAT'', ''CREWLET_GIT_TRAILER_RUN'', + ''CREWLET_GIT_TRAILER_WORK_ITEM''])' valueFrom: description: valueFrom is the Secret key the value is read from. @@ -1375,9 +1439,14 @@ spec: description: |- stopRunning ends the turn the seat is on at its next round when it is paused, rather than letting the turn finish; the stopped turn is not - run again. Only with paused true. Setting it on a seat already paused - stops the turn it is on then; unsetting it changes nothing, since a - stopped turn stays stopped. Crewlet's pause `stop_running`. + run again. A coding run the seat detached runs on, but every change + it would make is refused until the seat is resumed: a call it makes + through the tool bridge of any of the seat's tools not marked + read-only, and a push its box makes to the company's code host, which + the operator does not offer. Only with paused true. Setting it on a + seat already paused stops the turn it is on then, and the writes of + its runs; unsetting it changes nothing, since a stop lasts until the + seat is resumed. Crewlet's pause `stop_running`. type: boolean tokenBudget: description: |- @@ -1437,13 +1506,13 @@ spec: - message: contact and availability apply to Human seats only rule: self.kind != 'Agent' || (!has(self.contact) && !has(self.availability)) - message: 'a Human seat is never run by the engine: llm, tokenBudget, - workers, learningEnabled, mcp, sandbox, integrations, behavioralGuidelines, + workers, learningEnabled, mcp, sandbox, code, integrations, behavioralGuidelines, project, space, placement and paused apply to Agent seats only' rule: self.kind != 'Human' || !(has(self.llm) || has(self.tokenBudget) || has(self.workers) || has(self.learningEnabled) || has(self.mcp) - || has(self.sandbox) || has(self.integrations) || has(self.behavioralGuidelines) - || has(self.project) || has(self.space) || has(self.placement) || - has(self.paused)) + || has(self.sandbox) || has(self.code) || has(self.integrations) || + has(self.behavioralGuidelines) || has(self.project) || has(self.space) + || has(self.placement) || has(self.paused)) - message: 'pauseReason and stopRunning describe a pause: they apply only while paused is true' rule: (!has(self.pauseReason) && !has(self.stopRunning)) || (has(self.paused) @@ -1604,8 +1673,9 @@ spec: minLength: 1 type: string stopRunning: - description: stopRunning is whether the pause also ended the turn - the seat was on. + description: |- + stopRunning is whether the pause also ended the turn the seat was on, + and refuses its detached coding runs' writes until it is resumed. type: boolean type: object placement: diff --git a/docs/concepts/applying.md b/docs/concepts/applying.md index 7a70e72..96935ee 100644 --- a/docs/concepts/applying.md +++ b/docs/concepts/applying.md @@ -381,7 +381,17 @@ for a token no seat is bound to. A pause the operator took reads The engine keeps the reason a pause was taken with: a `pauseReason` changed while the seat stays paused shows from its next pause on. Setting `stopRunning` on a seat already paused ends the turn it is on then; unsetting -it changes nothing, since a stopped turn is not run again. +it changes nothing, since a stopped turn is not run again and a stop lasts +until the seat is resumed. + +A stop also reaches a coding run the seat detached, which outlives its turn +and runs on through the pause: until the seat is resumed, the engine refuses +every change the run would make, naming who stopped the seat. That is a call +it makes through the tool bridge of any of the seat's tools not marked +read-only, and a push its box makes to the company's code host, which the +operator does not offer. Its reads still answer. A pause without +`stopRunning` stops new work only, so a running job's writes land, as a turn +in flight finishes. The engine knows no seat it does not run, so a Seat created with `paused: true` is paused once the revision that adds it is applied: the @@ -448,7 +458,7 @@ Besides the conditions: and `/webhooks/github/`, set as soon as the Company has the vendor's Integration and a public address. While the engine holds it paused, `status.pause` says who paused it, when, why, and whether its - running turn was stopped. + running turn was stopped and its detached coding runs' writes refused. - **Unit**: its depth in the chart (`0` at the top; unset for a Unit outside it) and how many Seats are directly in it. - **Integration**: the company-wide address the vendor delivers to, for diff --git a/docs/concepts/company-document.md b/docs/concepts/company-document.md index d179c78..34eb81f 100644 --- a/docs/concepts/company-document.md +++ b/docs/concepts/company-document.md @@ -37,7 +37,7 @@ handle, and model providers and tool servers by name. | Object | Becomes | |---|---| -| `Company` | `name`, `mission`, `vision`, `policies`, `timezone`, `tracker`, `knowledge`, `token_budget`, `providers.embeddings`, `providers.sandbox`, `providers.llm_order`, `integrations.check_interval_seconds`, and one top-level key per `spec.engineConfig` entry | +| `Company` | `name`, `mission`, `vision`, `policies`, `timezone`, `tracker`, `knowledge`, `inbox`, `token_budget`, `providers.embeddings`, `providers.sandbox`, `providers.llm_order`, `integrations.check_interval_seconds`, and one top-level key per `spec.engineConfig` entry | | `EngineClass` | `providers.llm.` for each of its `platformLLMProviders`, when the Company sets `spec.llm.usePlatformProviders`; its `state_dir` is `/var/lib/crewlet/llm-cli-platform/`, on the data volume | | `LLMProvider` | `providers.llm.`; its `state_dir` is `/var/lib/crewlet/llm-cli/`, on the data volume | | `MCPServer` | one entry of `mcp_servers`, named by its `metadata.name` | @@ -121,6 +121,28 @@ plan, so a seat that needs a fallback names a chain. No provider may be named Each Seat's `spec.llm` becomes Crewlet's per-phase `llm` mapping, each chain naming providers by their key in `providers.llm`. +### Code access + +An agent Seat's `spec.code.access` becomes its role's `code.access` +(`read_only`, `review` or `full_access`), Crewlet's one tier for every code +host it drives: the seat's GitHub App is created with that tier's +permissions and mints every token at it. A seat that sets none runs at +`read_only`. The tier is the seat's, not its GitHub identity's, so +`spec.integrations.github` names only the app and the repositories it works +in, and the API refuses a tier on a Human seat, since a person acts as their +own identity on every host. Crewlet applies a tier that nothing caps and +warns of it (`code.access … grants nothing`): one on a seat with no +`spec.integrations.github`, or in a company with no GitHub `Integration` +([Read what the engine is worth telling you](../guides/troubleshooting.md#read-what-the-engine-is-worth-telling-you)). + +Crewlet's own code host, which keeps the company's git repositories in the +engine (its top-level `code` block: `native`, `repos`, `limits`), is not +offered in v1alpha1, and the API refuses a `spec.engineConfig` entry named +`code`. The host serves its repositories only from a node with the `git` +role and a cache of them on its volume, and no pod the operator runs has +either: a document that set it would declare repositories nothing can clone +or push. + ### Schedules A Schedule is written into the entry of what it targets: a Seat's role, or a @@ -244,8 +266,12 @@ opens the leaf listener seat pods join the event stream through Company has a pool yet: the class decides it, not the Company, so a company that adds its first pool restarts nothing, and a class that changes its policy restarts its companies' data pods. Like every port of Crewlet's -embedded broker, the listener accepts any connection that reaches it, with no -authentication and no TLS. +embedded broker, the listener admits a link only when it presents a key +derived from the company's keyring, which the operator writes into every +data pod's and seat pod's `crewlet.yaml`: each pod presents the active key +and accepts every key of the keyring it started with. The link carries no +TLS ([The fleet's broker admits only its own +nodes](https://github.com/crewlet/crewlet/blob/main/docs/guides/deployment.md#the-fleets-broker-admits-only-its-own-nodes)). The event stream is given three quarters of the data volume (`stream.store_max_bytes`), measured from the size the pod's volume claim @@ -258,13 +284,14 @@ last one is refused with `insufficient storage resources available`. A fresh node reserves what it keeps on the stream when it starts, and refuses to boot on a stream that cannot hold it ([the minimum room for a fresh node](https://github.com/crewlet/crewlet/blob/main/docs/guides/deployment.md#the-minimum-room-for-a-fresh-node)): -the ceilings of its seven state logs, 6979321856 bytes at their smallest, -and, with the class's `objects.backend` `NATS` (the default), the object -store's bucket beside them at a quarter of the stream, 9305762474 bytes in -all. So an EngineClass's `data.storage.size` is at least `12Gi`, whose three -quarters is 9 GiB, or `9Gi` for a class that keeps its files in S3. That is -the least that boots: the mailboxes and the coordination buckets grow into -what is left. Nor does Crewlet start on more +the ceilings of its eight state logs, 8053063680 bytes at their smallest +(the code host's log among them, which a node reserves whether or not the +company hosts code), and, with the class's `objects.backend` `NATS` (the +default), the object store's bucket beside them at a quarter of the stream, +10737418239 bytes in all. So an EngineClass's `data.storage.size` is at +least `14Gi`, whose three quarters is 10.5 GiB, or `10Gi` for a class that +keeps its files in S3. That is the least that boots: the mailboxes and the +coordination buckets grow into what is left. Nor does Crewlet start on more than 64 TiB, a guard against a unit mistake rather than a limit on the volume, so a volume larger than about 85 TiB gives the stream 64 TiB and leaves the rest to the store and its snapshots. diff --git a/docs/concepts/overview.md b/docs/concepts/overview.md index 48c8555..9a4bbe0 100644 --- a/docs/concepts/overview.md +++ b/docs/concepts/overview.md @@ -31,7 +31,7 @@ the [API reference](../reference/api.md). | Kind | Scope | What it is | Becomes in the company document | |---|---|---|---| | `EngineClass` | Cluster | A platform's defaults and guard rails: the engine image, the size of the data pod, how companies are exposed, limits, models the platform provides | Nothing directly; it shapes the pods, the routes and the limits of every Company that names it | -| `Company` | Namespace | The company itself, and the engine that runs it | `name`, `mission`, `vision`, `policies`, `timezone`, `tracker`, `knowledge`, `token_budget`, `providers.embeddings`, `providers.sandbox`, and the engine tuning blocks in `engineConfig` | +| `Company` | Namespace | The company itself, and the engine that runs it | `name`, `mission`, `vision`, `policies`, `timezone`, `tracker`, `knowledge`, `inbox`, `token_budget`, `providers.embeddings`, `providers.sandbox`, and the engine tuning blocks in `engineConfig` | | `Unit` | Namespace | One node of the org chart: a division, a team, a squad | One entry of `units`, nested under its parent | | `Seat` | Namespace | One seat: an AI agent, or a person agents work with, tied to their own accounts ([A person's accounts](../guides/integrations.md#a-persons-accounts)) | One entry of `roles`, inside its unit | | `LLMProvider` | Namespace | A model a company's seats can run on: a coding CLI the engine drives ([Running seats on a coding CLI](../guides/cli-agents.md)) | One entry of `providers.llm` | @@ -197,7 +197,11 @@ entry per top-level key of the document (`turn_engine`, `learning`, another field or object owns, and those are the only blocks that hold credentials, so engineConfig can never carry one. [Credential Positions](credentials.md) lists every credential position Crewlet has and the field that covers it, -including the ones v1alpha1 does not offer yet. +including the ones v1alpha1 does not offer yet. It refuses `code` too: +Crewlet's code host, which keeps the company's git repositories in the +engine, is not offered in v1alpha1, since no pod the operator runs has the +`git` role or the cache that serves them +([Code access](company-document.md#code-access)). --- diff --git a/docs/concepts/runtime.md b/docs/concepts/runtime.md index 8dd0a7f..4228aba 100644 --- a/docs/concepts/runtime.md +++ b/docs/concepts/runtime.md @@ -126,7 +126,7 @@ Two cases are refused rather than guessed at: | **Storage** | One volume claim per pod (`data-nimbus-data-0`) at `/var/lib/crewlet`, from the class's `data.storage`: the store, the event stream (three quarters of the volume, [sized from the claim](company-document.md#the-data-pods-crewletyaml)), and the store's native library cache (`TURSO_GO_CACHE_DIR`), so a restart does not extract it again. | | **Environment** | `CREWLET_NODE_ID`, the pod's name; `HOME`; `CREWLET_MCP_BRIDGE_URL`, the Company's public address, while an LLMProvider of the company runs a CLI in `mode: Agent` (see [Exposure](#exposure)); the OpenTelemetry variables when the class sets `telemetry`, and with them `CREWLET_SANDBOX_OTEL_RECEIVER_URL`, the Company's public address, while the Company has one (see [Exposure](#exposure)); then the class's `podTemplate.spec.env` and the Company's `runtime.env`, merged by name. No credential of the engine's own is an environment variable (see [What the engine is given](company-document.md#the-data-pods-crewletyaml)). All of it is the engine's: a stdio tool server the engine starts is handed only the part of it [Crewlet's allowlist](https://github.com/crewlet/crewlet/blob/main/docs/guides/tools-and-mcp.md#what-a-stdio-servers-environment-is) names, the proxy and TLS variables among it, a coding CLI that part and the variables its Crewlet profile passes through, and a sandbox none ([Stdio and HTTP](../guides/tool-servers.md#stdio-and-http)). | | **Ports** | 8080 (`http`), the API, the dashboard and the probes; 8081 (`public`), the public listener: the webhooks, and a remote sandbox's `/otlp/` and `/mcp/`; 8082 (`artifacts`), the artifact listener, while the company has an artifact origin; and 7422 (`leaf`), the leaf listener seat pods join the event stream through, while the class's `seatPools.policy` is `Allowed`. The headless Service `nimbus-data` names 8080 and 7422; `nimbus-engine` names 8080, 8081 and 8082. | -| **Probes** | `/health` for startup and liveness, `/ready` for readiness. Startup allows at least the class's `data.startupSeconds`, counted from the container's start, or, unset, 1,636 seconds (1,776 when the class keeps files in S3) plus six seconds per GiB of the data volume. The first figure is every bound the engine puts on a data pod's boot, laid end to end, and the API refuses a `startupSeconds` below it: its waits for its own broker and store, never for a peer. One of them grows with what the node holds and is left to the second figure, as the recovery is: the claim of each batch of a seat pod's events the node kept and could not settle, 15 s a batch. The second is for what the engine waits out unbounded: it recovers what it holds on the volume before `/health` answers `200`, which takes longer the more it holds, and a pod restarted mid-recovery starts it again. What the company configures does not count: `/health` answers once the recovery is over, and only then does the engine read the company's secrets, start its tool servers and restore its CLIs' logins, while `/ready` answers `503` `starting` until it serves. Liveness allows a minute of failures before a restart, longer than the 45 seconds in which the engine's watchdog ends a wedged process itself. `/health` answers while the engine drains, so liveness never kills a pod finishing its turns. | +| **Probes** | `/health` for startup and liveness, `/ready` for readiness. Startup allows at least the class's `data.startupSeconds`, counted from the container's start, or, unset, 1,461 seconds (1,601 when the class keeps files in S3) plus six seconds per GiB of the data volume. The first figure is every bound the engine puts on a data pod's boot, laid end to end, and the API refuses a `startupSeconds` below it: its waits for its own broker and store, never for a peer. One of them grows with what the node holds and is left to the second figure, as the recovery is: the claim of each batch of a seat pod's events the node kept and could not settle, 15 s a batch. Each read of a state log's state or record counts at thirty seconds: the engine asks one again while the broker answers none, for thirty seconds and no longer, so a boot whose broker never answers ends by itself inside the window rather than waiting for the probe to restart it. The second is for what the engine waits out unbounded: it recovers what it holds on the volume before `/health` answers `200`, which takes longer the more it holds, and a pod restarted mid-recovery starts it again. What the company configures does not count: `/health` answers once the recovery is over, and only then does the engine read the company's secrets, start its tool servers and restore its CLIs' logins, while `/ready` answers `503` `starting` until it serves. Liveness allows a minute of failures before a restart, longer than the 45 seconds in which the engine's watchdog ends a wedged process itself. `/health` answers while the engine drains, so liveness never kills a pod finishing its turns. | | **Stopping** | A five-second `preStop` sleep, so the endpoint is gone from the Services and Gateways before the engine drains, and the class's `data.terminationGracePeriodSeconds` (default five minutes) for the drain: running turns finish, waiting ones are handed back. | | **Updates** | `RollingUpdate` with `maxUnavailable: 1`, `podManagementPolicy: Parallel`. | | **Scheduling** | The class's `podTemplate`: labels, annotations, node selector, tolerations, affinity, topology spread, runtime and priority class; and its `data.resources`. | @@ -194,7 +194,7 @@ flowchart LR | Environment | `CREWLET_NODE_ID`, the pod's name; `HOME`; `TURSO_GO_CACHE_DIR` on the scratch volume; the OpenTelemetry variables and `CREWLET_SANDBOX_OTEL_RECEIVER_URL` as on a data pod (never `CREWLET_MCP_BRIDGE_URL`); the class's and the Company's variables, as on a data pod | the same, and `CREWLET_SEAT: cy` | | Storage | one `emptyDir` at `/var/lib/crewlet`, of the class's `seatPools.defaults.scratchSizeLimit` (2Gi unless set), holding the scratch store, the store's native library, `/tmp` and `HOME`: the limit bounds what the processes the engine runs write too | the same | | Stopping | the pool's `terminationGracePeriodSeconds`, else the class's default, else five minutes: running turns finish, waiting ones are handed back, and the seats are released with their mailboxes for another pod to claim | the same | -| Port and probes | 8080, the probes alone: `/health` for startup (1,142 seconds, 1,192 with an S3 object store) and liveness, `/ready` for readiness | the same | +| Port and probes | 8080, the probes alone: `/health` for startup (1,137 seconds, 1,187 with an S3 object store) and liveness, `/ready` for readiness | the same | | Updates | `RollingUpdate`, `maxUnavailable: 1`, `podManagementPolicy: Parallel`, `minReadySeconds: 10` | the same | | Resources | the class's `seatPools.defaults.resources`: the class sizes every seat pod, as it sizes the data pods, and a Company chooses only how many its pools run, within `maxPods` | the same | @@ -220,8 +220,8 @@ pods publishes no port. The probes are the data pod's: `/health` answers while the process is alive, through a drain, and liveness restarts it after a minute of failures, the backstop for a process too wedged for the engine's own watchdog to end -(with code 75) within 45 seconds. Startup allows at least 1,142 seconds, -about 19 minutes, or 1,192 when the class keeps files in S3: every bound +(with code 75) within 45 seconds. Startup allows at least 1,137 seconds, +about 19 minutes, or 1,187 when the class keeps files in S3: every bound the engine puts on a seat pod's boot, laid end to end. Most of them wait on the data pod: a minute for the leaf link to reach it and its JetStream to answer, five for the streams the seat pod uses, five more for the @@ -615,10 +615,12 @@ Two more policies come with seat pools: seat pod), and for as long as any data pod that runs, or is stopping, was started from a template that opened it (the pod's annotation `crewlet.ai/leaf-listener: open`), however long the rollout that closes it - takes or is held back. Crewlet's leaf listener accepts any - connection, with no authentication and no TLS, and whoever joins it reads - and writes the company's whole event stream. It selects the data pods and - admits port 7422 only from pods labelled as this company's seat pods. Under + takes or is held back. Crewlet's leaf listener admits a link only on a + key derived from the company's keyring, but it encrypts nothing, and + whoever joins it reads and writes the company's whole event stream, so + the policy keeps every other pod from reaching it at all: it selects the + data pods and admits port 7422 only from pods labelled as this company's + seat pods. Under `None` it also admits ports 8080, 8081 and 8082 from anywhere, which keeps the API, the public listener and the artifact listener as open as they are without a policy and closes only the broker. @@ -634,11 +636,13 @@ only while no policy of anyone else's selects them. > **Caution: the namespace is the trust boundary.** A policy selects pods by > their labels, and whoever can create a pod in the namespace can give it a -> seat pod's labels and join the company's event stream; they can also mount -> the company's keyring Secret into it and open its secret store, which no -> policy governs. Companies that must not read each other's streams and -> secrets need namespaces of their own, or a namespace where nobody but the -> operator creates pods. +> seat pod's labels and mount the company's keyring into it, which is all a +> seat pod needs to join the company's event stream: `-keyring` +> holds it, and so do `-bootstrap` and every +> `-bootstrap--`. The keyring also opens the company's +> secret store, and no policy governs either. Companies +> that must not read each other's streams and secrets need namespaces of +> their own, or a namespace where nobody but the operator creates pods. Two more limits hold for every NetworkPolicy, on most CNIs (Calico and Cilium among them): traffic from the node itself, and from `hostNetwork` pods diff --git a/docs/getting-started.md b/docs/getting-started.md index 7fddf7e..9f7e65c 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -94,7 +94,7 @@ gives each company the smallest data volume the API admits. > **Watch for disk space.** The engine reserves three quarters of that volume > for its event stream, and kind's provisioner puts it on the node's disk -> without enforcing the size: the Docker host needs about 10 GiB free for each +> without enforcing the size: the Docker host needs about 12 GiB free for each > company, or its pod never becomes ready. ```bash @@ -112,7 +112,7 @@ spec: cpu: 100m memory: 256Mi storage: - size: 12Gi + size: 14Gi exposure: webhooks: mode: None diff --git a/docs/guides/credentials.md b/docs/guides/credentials.md index 7522c22..fbbfda0 100644 --- a/docs/guides/credentials.md +++ b/docs/guides/credentials.md @@ -117,6 +117,19 @@ Variables and files the sandbox sets up for every run (`sandbox.setup[].env`, `sandbox.setup[].files`) and a Seat's own sandbox variables (`Seat.spec.sandbox.env`) take `valueFrom.secretKeyRef` for every entry. +None of those variables, a Company's step's, a Seat's step's or the Seat's +own, may be one Crewlet sets in a coding run itself: `CREWLET_GIT_TOKEN` and +`CREWLET_GIT_URL`, a run's access to the company's own code host, and +`CREWLET_GIT_TRAILER_SEAT`, `CREWLET_GIT_TRAILER_RUN` and +`CREWLET_GIT_TRAILER_WORK_ITEM`, what its commit hook writes into every +commit's trailers (Crewlet's +[Repositories on this company's code host](https://github.com/crewlet/crewlet/blob/main/docs/concepts/code-sandbox.md#repositories-on-this-companys-code-host)). +Crewlet refuses a company document that sets one, and the API refuses it at +the variable's name. The operator does not offer the code host, so no run of +its companies is given those variables, but the names are Crewlet's all the +same: a variable of your own, such as a token for another git host, takes +another name. + A file's `path` is where the seat's E2B box writes it. Name it relative to the box's home, `/home/user` (`.npmrc`), or under `~/` (`~/.npmrc`); a `..` that climbs out of the home lands where it resolves. An absolute path works @@ -201,9 +214,10 @@ kubectl create secret generic lead-github-app -n acme \ ```yaml spec: # Seat lead + code: + access: Review # how much the app may do; ReadOnly when unset integrations: github: - tier: Review repos: [acme/widgets] app: id: 1234567 @@ -212,7 +226,10 @@ spec: # Seat lead credentialsSecretRef: {name: lead-github-app} # keys privateKey, webhookSecret ``` -The webhook URL for the app is `status.integrations.github.webhookURL`. +The webhook URL for the app is `status.integrations.github.webhookURL`. How +much the app may do is the seat's `spec.code.access`, not a field of its +GitHub identity: Crewlet keeps one tier for every code host it drives, and +creates the app with that tier's permissions. ### The company's GitHub organization @@ -457,6 +474,19 @@ seat pod of the company, each restarting until the key is back in Adding a key to the Secret is safe (the pod rolls onto a keyring that opens everything it did); removing one is not. +Adding a key and making it active (`crewlet.ai/active-key-id`) in one edit +is safe as well. A seat pod joins the company's event stream presenting a +key derived from its keyring's active key, and a data pod admits only the +keys of the keyring it started with, but the operator moves the seat pods +onto a new keyring only once every data pod is ready on it: until then they +present the key that was active, which the data pods still hold, and after +it the new one, which every data pod already holds. An edit that removes the +key the seat pods present, making another active, does cut them off for a +while, when the data pods boot on it at all: the data pods roll onto a +keyring without that key and refuse each seat pod until it, too, has rolled +onto the new keyring +([Seat pods do not run](troubleshooting.md#seat-pods-do-not-run)). + ## When a credential does not arrive | You see | It means | diff --git a/docs/guides/exposure.md b/docs/guides/exposure.md index 999f471..bbeef73 100644 --- a/docs/guides/exposure.md +++ b/docs/guides/exposure.md @@ -80,6 +80,19 @@ say so. What the public listener serves depends on what the engine is given: operator gives the engine the public address as the receiver's (`CREWLET_SANDBOX_OTEL_RECEIVER_URL`). +Crewlet reads both variables at every start, as a sandbox's URL parser (the +WHATWG URL Standard's) will, and refuses to start on a value that parser +cannot read: a host with an IPv6 zone, with a `%`, `<`, `>` or `]` in its name +(an unfilled placeholder such as `` among them), or whose last +label is a number and that is no IPv4 address. So the API refuses such a +`publicBaseURL`, and takes an IPv4 address there as four decimal octets alone, +and refuses a class's `webhooks.hostname` whose last label is a number, such +as `{{uid}}.hooks.123`. A last label a variable fills is a number only for +some Companies ([Reach the API from elsewhere](#reach-the-api-from-elsewhere) +says which), and while the operator hands such a Company's address to +sandboxes it holds that Company's engine back, with `RuntimeReady=False +AddressRefused`, rather than start one that refuses to boot. + The address appears in the Company's `status.endpoints.publicBaseURL`, and each Seat's and Integration's status carries the exact URL to register with the vendor. @@ -592,8 +605,10 @@ A class that allows seat pools (`seatPools.policy: Allowed`) also has the operator create two more policies per company, under either mode: - `-broker` admits the data pods' leaf listener (port 7422) from the - company's own seat pods and from nothing else, since Crewlet's broker port - takes any connection without authentication or TLS. Under `None` it also + company's own seat pods and from nothing else. Crewlet's broker admits a + link there only on a key derived from the company's keyring, but it + encrypts nothing, and the policy keeps every other pod from reaching the + port at all. Under `None` it also admits ports 8080, 8081 and 8082 from anywhere, so the data pods stay as open as they are without a policy. - `-seats`, while the company runs seat pods, admits no connection to diff --git a/docs/guides/installing.md b/docs/guides/installing.md index 9d57dd9..cd7ba40 100644 --- a/docs/guides/installing.md +++ b/docs/guides/installing.md @@ -19,8 +19,15 @@ page installs it on a cluster you run companies on. The - A CNI that enforces NetworkPolicy, for the isolation around each engine ([Exposure and network policy](exposure.md#network-policy)). - A default StorageClass, or the one your EngineClasses name, that provisions - `ReadWriteOnce` volumes of at least 12Gi (9Gi for an EngineClass that keeps - its files in S3). + `ReadWriteOnce` volumes of at least 14Gi (10Gi for an EngineClass that keeps + its files in S3). The floor follows what a fresh engine node reserves on its + event stream, so an operator that runs a newer engine can raise it: an + EngineClass stored below it keeps its size, every Company created under it + gets a volume no engine boots on, and the API refuses a change to its spec + until its size is raised (with its files in S3, only a change of its size + that leaves it below), so raise its `data.storage.size`. A company already + running on such a volume needs its own claim grown + ([The data pod is not ready](troubleshooting.md#the-data-pod-is-not-ready)). - A way to take each company's backups: the operator takes none, and the engine trims its logs only behind a recent one, refusing writes once they are full ([Backups](../concepts/runtime.md#backups)). diff --git a/docs/guides/integrations.md b/docs/guides/integrations.md index 4f9f8d2..de4494c 100644 --- a/docs/guides/integrations.md +++ b/docs/guides/integrations.md @@ -378,7 +378,10 @@ Slack has no company-wide credential: each agent seat is its own app, in its Seat's `spec.integrations.slack`. GitHub's Integration holds the webhook secret of the company-wide route and an optional organization token, and each agent seat is its own GitHub App, in `spec.integrations.github`. -[Supplying credentials](credentials.md#an-agents-slack-app) shows both. +[Supplying credentials](credentials.md#an-agents-slack-app) shows both. How +much a seat's app may do in a repository is the Seat's `spec.code.access` +(`ReadOnly`, the default, `Review` or `FullAccess`), Crewlet's one tier for +every code host it drives. A Slack agent posts through the `slack` tool server, `slack-mcp-server`, on its app's bot token, which its Seat names a second time for the server; the diff --git a/docs/guides/multi-tenancy.md b/docs/guides/multi-tenancy.md index 6b79003..2016392 100644 --- a/docs/guides/multi-tenancy.md +++ b/docs/guides/multi-tenancy.md @@ -139,8 +139,10 @@ apart: has stopped its engine. Within a namespace nothing keeps two Companies apart: whoever can create a -pod there can mount either's keyring Secret, or label the pod as either's -seat pod and join its event stream, which no NetworkPolicy prevents. Run one +pod there can mount either's keyring, which opens its secret store and is +the key its event stream admits (`-keyring` holds it, and so does +each of the Company's bootstrap Secrets), and label the pod as either's seat +pod to reach that stream, which no NetworkPolicy prevents. Run one Company per tenant namespace, and let nobody but the operator create pods in it ([Network policy](../concepts/runtime.md#network-policy)). diff --git a/docs/guides/tool-servers.md b/docs/guides/tool-servers.md index 0b9591a..ac9b8cc 100644 --- a/docs/guides/tool-servers.md +++ b/docs/guides/tool-servers.md @@ -195,8 +195,10 @@ has the engine's side. ## The image needs the server -Crewlet's image is Debian with `ca-certificates`, `git`, `tini` and the -engine: no Python, Node.js, `uv`, `npm` or `curl`. Its +Crewlet's image is Debian with `ca-certificates`, `tini`, the engine and +git, which the image builds from the source release Crewlet pins +(`scripts/git-pin.env`) and installs under `/usr/local`, in place of +Debian's `git` package: no Python, Node.js, `uv`, `npm` or `curl`. Its [Dockerfile](https://github.com/crewlet/crewlet/blob/main/Dockerfile) says so on purpose: a stdio server whose runtime is not there needs a derived image. diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index ca02ac3..1e5d11b 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -77,7 +77,7 @@ with your last change yet. | | False | `SecretNotFound` | A Secret the pods are given values from is missing: one the class references (telemetry headers, `podTemplate.spec.env`) in the platform's namespace, or the Company's own object-store keys (`spec.objects.credentialsSecretRef`). The pod is held as it is. | | | False | `APITokensInvalid` | `-api-tokens`, as a platform supplied it, lacks a token or holds one shorter than 43 characters or outside base64, base64url and hex. Repair it, or delete it to have the operator generate it. The pod is held as it is. | | | False | `InsecureSignInNotAcknowledged` | The class lets people sign in with a password alone (`spec.signIn.secondFactor: Optional`), and the Company is reached at an address Crewlet does not read as loopback, which the message names: Crewlet refuses to start so without the class's `spec.signIn.acceptInsecure: true`. The platform acknowledges it on the class or requires a second factor again ([Two-step verification](people.md#two-step-verification)). The pod is held as it is. | -| | False | `AddressRefused` | An address the engine would be told it is reached at is one Crewlet refuses to start on; the message names it, its key in `crewlet.yaml` and where it came from. In practice it is the class's API route: its `api.hostname` template's last label holds `{{namespace}}` or `{{name}}`, and the Company's namespace is a number, such as `4711` or `0x1f`, or its namespace or name makes one with the characters beside it (`0x{{name}}` for a Company named `cafe`), so the host's last label is a number, which a browser reads as an IPv4 address. The platform ends the template in a domain of its own (`{{name}}.{{namespace}}.crewlet.example.com`), or the Company runs in a namespace of another name ([Reach the API from elsewhere](exposure.md#reach-the-api-from-elsewhere)). An address the Company states is refused by the API when it is written instead, unless it was admitted before the API refused it: the message then says to correct the Company's field. An artifact origin that meets the dashboard's is refused here too: give artifacts a domain of their own. The pod is held as it is. | +| | False | `AddressRefused` | An address the engine would be told it is reached at, or would hand its sandboxes (`CREWLET_MCP_BRIDGE_URL`, `CREWLET_SANDBOX_OTEL_RECEIVER_URL`), is one Crewlet refuses to start on; the message names it, its key in `crewlet.yaml` or its variable, and where it came from. In practice it is a route of the class: its API route, or its webhooks route while the engine hands that address to sandboxes, whose host name template's last label holds `{{namespace}}` or `{{name}}`, and the Company's namespace is a number, such as `4711` or `0x1f`, or its namespace or name makes one with the characters beside it (`0x{{name}}` for a Company named `cafe`), so the host's last label is a number, which a browser and a sandbox read as an IPv4 address. The platform ends the template in a domain of its own (`{{name}}.{{namespace}}.crewlet.example.com`), or the Company runs in a namespace of another name ([Reach the API from elsewhere](exposure.md#reach-the-api-from-elsewhere)). An address the Company states is refused by the API when it is written instead, unless it was admitted before the API refused it: the message then says to correct the Company's field. An artifact origin that meets the dashboard's is refused here too: give artifacts a domain of their own. The pod is held as it is. | | `Exposed` | True | `RouteAccepted` | Every route the class asks for was accepted. When the message adds that the engine is told people reach it at `http://localhost:8080`, every link it gives a person, an invitation's included, leads there through a port-forward: set the Company's `spec.exposure.dashboardBaseURL`, or route the API ([Where people reach the dashboard](exposure.md#where-people-reach-the-dashboard)). | | | True | `NoRoutes` | The class routes nothing (webhooks `None` or `External`, API `ClusterInternal`, artifacts `None`). The message says the same about the dashboard's address as under `RouteAccepted`. | | | False | `RouteNotAccepted` | A Gateway has not attached the HTTPRoute, or the Ingress controller has not published an address. The message carries the controller's own reason; a Gateway listener's `allowedRoutes` that does not admit the tenant's namespace is the usual cause. | @@ -207,28 +207,48 @@ kubectl logs acme-data-0 -n acme | grep -E 'api_bound|engine_recovered|api_liste - **`the broker refused to reserve the … log's ceiling`**, ending `insufficient storage resources available`: the engine could not reserve what a fresh node keeps on its event stream. The stream is given three - quarters of the data volume, and a fresh node reserves at least 9305762474 - bytes of it, or 6979321856 when the class keeps its files in S3; the + quarters of the data volume, and a fresh node reserves at least 10737418239 + bytes of it, or 8053063680 when the class keeps its files in S3; the message names the figure it needed and breaks it down. The class's - `data.storage.size` admits no less than `12Gi` (`9Gi` with the files in + `data.storage.size` admits no less than `14Gi` (`10Gi` with the files in S3), so this is a company whose volume was made smaller: under an older floor, or before an engine that keeps more on its stream. [Grow its volume](../concepts/runtime.md#growing-a-companys-data-volume). + A company made at the floor before Crewlet's code host (`12Gi`, or `9Gi` + with the files in S3) is one: the engine that adds the host's log, + `CREWLET_CODE_LOG`, reserves 1 GiB for it beside the logs the node already + holds, whether or not the company hosts code, and finds no room + (`the broker refused to reserve the code log's ceiling`); grow its volume + to the floor. Raise every EngineClass stored below the floor as well: each + Company created under it gets a volume of its size, on which no fresh node + boots, and the API refuses a change to the class's spec until its + `data.storage.size` reaches the floor (with the files in S3, only a change + of the size that leaves it below). - **`config_apply_failed`**: the revision was stored but this node could not start it; the message says why. The engine keeps running the previous revision, and the Company reports `ConfigApplied=False ApplyFailed` (see [The engine did not apply a revision](#the-engine-did-not-apply-a-revision)). - **The pod restarts** (`kubectl get pod` shows restarts): the engine refused its own configuration at boot, and the first lines of the previous log say - why (`kubectl logs acme-data-0 -n acme --previous`). + why (`kubectl logs acme-data-0 -n acme --previous`). Or its broker did not + answer a request the boot waits on: the engine gives each up within a + bound the startup window counts (a read of a state log, which it asks + again while nobody answers and logs nothing for, after thirty seconds), + and the previous log ends on `crewlet:` and the request nobody answered, + such as a state log's `read "…"'s state`. What the broker itself reported + is in the previous log under `queue.nats.server` + (`kubectl logs acme-data-0 -n acme --previous | grep queue.nats.server`; + [The embedded broker's own logs](https://github.com/crewlet/crewlet/blob/main/docs/guides/deployment.md#the-embedded-brokers-own-logs)). - **The pod restarts, its events saying `Startup probe failed`, and the previous log has no `engine_recovered` line**: the recovery of what the volume holds took longer than the startup probe allows, and every restart - starts it again. The window is 1,636 seconds (1,776 when the class keeps + starts it again. The window is 1,461 seconds (1,601 when the class keeps files in S3) plus six seconds per GiB of the volume; a disk that reads slower than that needs a longer one. Set the class's `data.startupSeconds` (up to a day), which rolls the data pods once, and let the next boot - finish. + finish. A broker that does not answer is not this case: the window counts + every bound the engine puts on a boot's requests, so the engine ends such + a boot itself, as the row above says. - **The log has `engine_recovered` and no `api_listening` after it**: the engine has recovered what the volume holds and is starting what the company configures. It reads the company's secrets first, and while the secret @@ -315,6 +335,21 @@ the company's own seat pods to port 7422 of its data pod, and a policy of yours that selects the data pod or the seat pods may refuse what it admits by adding isolation of its own. +A seat pod's leaf can also reach the data pod and be refused. Its log says +the same `made no link to any member`, naming the `secrets.keys` id it +presents, and the data pod logs `broker_peer_refused` with `listener` `leaf` +and `presented` `unknown`, at most once a minute; that line names the leaf's +public nkey rather than the id. A seat pod presents a key derived from its +keyring's active key, and a data pod admits only the keys of the keyring it +started with. The operator moves the seat pods onto a new keyring only once +every data pod is ready on it, so this follows an edit of `-keyring` +that removed the key the seat pods presented: the data pods rolled onto a +keyring without it, and each seat pod is refused until it rolls onto the new +keyring after them. A key added and made active in one edit does not cause +it. The data pods do not boot at all while the secret store holds a value +sealed under the removed key +([The engine's own Secrets](credentials.md#the-engines-own-secrets)). + After a pool is removed, its StatefulSet keeps running until a revision that pins no seat to it runs: `SeatsPlaced` names it, and the Company's `status.config.placement` says what the running revision pins. @@ -684,7 +719,9 @@ a routable address sends passwords unencrypted. An `https` address with no proxy listed shows every caller as the proxy. A class may let people sign in with a password alone at a routable address, which it acknowledged. A company may have no human seat anyone can sign in to, or a token ceiling another -leaves idle. None of these stops the engine. It says so in two places: +leaves idle. A seat's `spec.code.access` may grant nothing, for want of a +GitHub `Integration` in the company or of `spec.integrations.github` on the +seat. None of these stops the engine. It says so in two places: - **The Company's events**: the engine's answer to each revision the operator writes, as an `EngineWarnings` event (`The engine runs revision … with diff --git a/docs/reference/api.md b/docs/reference/api.md index a4a2c30..0114b2a 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -56,7 +56,7 @@ _Appears in:_ | `mode` _[APIExposureMode](#apiexposuremode)_ | mode is how the API is reached. | ClusterInternal | Enum: [ClusterInternal Gateway Ingress]
Optional: \{\}
| | `gateway` _[GatewayExposure](#gatewayexposure)_ | gateway is the Gateways to attach to, for mode Gateway. | | Optional: \{\}
| | `ingress` _[IngressExposure](#ingressexposure)_ | ingress is the Ingress settings, for mode Ingress. | | Optional: \{\}
| -| `hostname` _string_ | hostname is the API's host name, as a template with the variables and
rules of webhooks.hostname, and one rule more.
It is also where a browser reaches the engine (the Company's
status.endpoints.dashboardBaseURL, unless the Company states its own),
and an origin the engine accepts a person's write from, and Crewlet
refuses to start on a browser's address whose host ends in a label
that is a number: a browser reads such a host as an IPv4 address and
rewrites it before it sends it. So its last label may not be one
(\{\{uid\}\}.123, \{\{uid\}\}.0x1f), where webhooks.hostname's may. A last
label a variable fills is a number only for some Companies: one whose
namespace is a number (4711, 0x1f) under \{\{name\}\}.\{\{namespace\}\}, or
whose namespace or name makes one with the template's own characters
beside it (\{\{uid\}\}.0x\{\{name\}\} for a Company named cafe; a Company's
name begins with a letter, so it is never a number alone). That
Company's engine is held back, with RuntimeReady=False AddressRefused,
rather than started to refuse. Ending the template in a domain of your
own, as in \{\{name\}\}.\{\{namespace\}\}.crewlet.example.com, leaves it to no
Company. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| +| `hostname` _string_ | hostname is the API's host name, as a template with the variables and
rules of webhooks.hostname, and one rule more.
It is also where a browser reaches the engine (the Company's
status.endpoints.dashboardBaseURL, unless the Company states its own),
and an origin the engine accepts a person's write from, and Crewlet
refuses to start on a browser's address whose host ends in a label
that is a number: a browser reads such a host as an IPv4 address and
rewrites it before it sends it. So its last label may not be one
(\{\{uid\}\}.123, \{\{uid\}\}.0x1f), as webhooks.hostname's may not. A last
label a variable fills is a number only for some Companies: one whose
namespace is a number (4711, 0x1f) under \{\{name\}\}.\{\{namespace\}\}, or
whose namespace or name makes one with the template's own characters
beside it (\{\{uid\}\}.0x\{\{name\}\} for a Company named cafe; a Company's
name begins with a letter, so it is never a number alone). That
Company's engine is held back, with RuntimeReady=False AddressRefused,
rather than started to refuse. Ending the template in a domain of your
own, as in \{\{name\}\}.\{\{namespace\}\}.crewlet.example.com, leaves it to no
Company. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| #### APIExposureMode @@ -379,6 +379,27 @@ _Appears in:_ | `valueFrom` _[NamespacedSecretValueSource](#namespacedsecretvaluesource)_ | valueFrom reads the value from a Secret key in a namespace the
platform controls. | | Optional: \{\}
| +#### CodeAccessTier + +_Underlying type:_ _string_ + +CodeAccessTier is how much a seat may do in a repository. It maps onto +Crewlet's role code.access (ReadOnly is `read_only`, Review is `review`, +FullAccess is `full_access`). + +_Validation:_ +- Enum: [ReadOnly Review FullAccess] + +_Appears in:_ +- [SeatCode](#seatcode) + +| Field | Description | +| --- | --- | +| `ReadOnly` | | +| `Review` | | +| `FullAccess` | | + + #### CodingAgent _Underlying type:_ _string_ @@ -462,7 +483,7 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `publicBaseURL` _string_ | publicBaseURL is the HTTPS address vendors deliver webhooks to, and a
remote sandbox calls the engine back on, overriding the one the class's
webhooks.hostname derives. Required when the class's webhooks.mode is
External, where something outside the operator (a relay, an API
gateway) forwards it to the engine's public listener. A trailing '/'
is dropped, as Crewlet drops it. Crewlet's api.public.external_url,
in the data pod's crewlet.yaml, which the engine reads only at start:
changing it restarts the data pod. Its host is a name, an IPv4
address or an IPv6 address in brackets, followed by at most one port,
1 to 65535: Crewlet refuses to start on any other, but for an IPv6
address whose zone holds a ']' (https://[fe80::1%25eth0]]/), which it
starts on and no vendor reaches. The API judges the host and the port
itself rather than leaving them to the API server's URL parser, which
is Go's net/url as the API server was built and is run: it takes such
a zone, and under Go's GODEBUG setting urlstrictcolons=0
https://h.example:80:80 too. | | MaxLength: 2048
MinLength: 1
Optional: \{\}
| +| `publicBaseURL` _string_ | publicBaseURL is the HTTPS address vendors deliver webhooks to, and a
remote sandbox calls the engine back on, overriding the one the class's
webhooks.hostname derives. Required when the class's webhooks.mode is
External, where something outside the operator (a relay, an API
gateway) forwards it to the engine's public listener. A trailing '/'
is dropped, as Crewlet drops it. Crewlet's api.public.external_url,
in the data pod's crewlet.yaml, which the engine reads only at start:
changing it restarts the data pod. Its host is a name, an IPv4
address or an IPv6 address in brackets, followed by at most one port,
1 to 65535: Crewlet refuses to start on any other, but for an IPv6
address whose zone holds a ']' (https://[fe80::1%25eth0]]/), which it
starts on and no vendor reaches. The API judges the host and the port
itself rather than leaving them to the API server's URL parser, which
is Go's net/url as the API server was built and is run: it takes such
a zone, and under Go's GODEBUG setting urlstrictcolons=0
https://h.example:80:80 too.
The engine also hands it to its sandboxes, as its tool bridge's address
(CREWLET_MCP_BRIDGE_URL) while an LLMProvider of the company runs in
mode Agent, and as its telemetry receiver's
(CREWLET_SANDBOX_OTEL_RECEIVER_URL) while the class sends telemetry to
a collector, and refuses to start on one the URL parser of a sandbox's
coding agent refuses, the WHATWG URL Standard's. So its host may not
hold an IPv6 zone or another '%', a '<', a '>' or a ']' outside an
IPv6 address's brackets, and a host whose last label is a number must
be an IPv4 address written as four decimal octets: that parser reads
such a host as an address, and Crewlet refuses one that is none. | | MaxLength: 2048
MinLength: 1
Optional: \{\}
| | `dashboardBaseURL` _string_ | dashboardBaseURL is where a browser reaches the engine: its dashboard,
and the sign-in of the company's people. It overrides the address of
the class's API route: set it when the class routes no API, or people
reach it through something in front of the route (a single sign-on
proxy, say). Crewlet's api.external_url, in the data pod's
crewlet.yaml, which the engine reads only at start: changing it
restarts the data pod. The engine takes from it the origin a person's
write must come from, whether the session cookie is Secure (https)
and its scope, and the base of every link a person follows: a link an
agent composes, an invitation (/dashboard#/invite/...), a password
link. An address other than the route's must serve the engine, its
/dashboard and its /auth routes included. The route, when the class
has one, stays an origin the engine accepts. It is never the public
address: the public listener serves no dashboard. A trailing '/' is
dropped.
Crewlet compares the origin of a person's write with this address as a
browser spells it, and refuses to start on an address whose host a
browser rewrites before it sends it, so the API refuses it too: a host
outside ASCII, as it is or percent-encoded (write its xn-- form,
https://xn--bcher-kva.example for https://bücher.example), a host
whose last label is a number and that is not an IPv4 address of four
decimal octets (127.1, 2130706433, 0x7f.0.0.1, 192.168.010.1,
crewlet.123: write 127.0.0.1, or a name that ends in a word), and an
IPv6 address with a zone ([fe80::1%25eth0]). Its host is a name, an
IPv4 address or an IPv6 address in brackets, followed by at most one
port, 1 to 65535 (http://[::1]:8080, not http://::1:8080), which the
API judges itself, as for publicBaseURL. | | MaxLength: 2048
MinLength: 1
Optional: \{\}
| | `artifactsBaseURL` _string_ | artifactsBaseURL is where a browser reaches the artifact origin,
ahead of the class's artifacts route: a tunnel, or http://localhost:8082
for a browser that port-forwards it. Crewlet's api.artifacts.external_url;
changing it restarts the data pod. https, or http on localhost or
127.x.x.x; \{artifact\} at the start of the host gives each artifact a host
of its own. It may not be the dashboard's or the public listener's
origin, nor a parent of either's host (RuntimeReady=False AddressRefused). | | MaxLength: 2048
MinLength: 1
Optional: \{\}
| @@ -586,12 +607,13 @@ _Appears in:_ | `timeZone` _string_ | timeZone is the company's clock, an IANA time zone such as
Europe/Berlin: where its days begin for budgets, due dates and
schedules. Unset is UTC. Crewlet's `timezone`. | | MaxLength: 64
MinLength: 1
Pattern: `^[A-Za-z][A-Za-z0-9_+-]*(/[A-Za-z0-9_+-]+)*$`
Optional: \{\}
| | `tracker` _[TrackerSettings](#trackersettings)_ | tracker is which work tracker the company runs. | | MinProperties: 1
Optional: \{\}
| | `knowledge` _[KnowledgeSettings](#knowledgesettings)_ | knowledge is which knowledge base the company runs. | | MinProperties: 1
Optional: \{\}
| +| `inbox` _[InboxSettings](#inboxsettings)_ | inbox is the company's policy over a person's inbox. | | MinProperties: 1
Optional: \{\}
| | `tokenBudget` _[TokenBudget](#tokenbudget)_ | tokenBudget caps the tokens all seats together may spend per window.
Crewlet's `token_budget`. | | MinProperties: 1
Optional: \{\}
| | `llm` _[CompanyLLMSettings](#companyllmsettings)_ | llm is how the company chooses among its model providers. | | MinProperties: 1
Optional: \{\}
| | `embeddings` _[EmbeddingSettings](#embeddingsettings)_ | embeddings is the provider vectors are computed with. Unset leaves
search keyword-only. Crewlet's providers.embeddings. | | Optional: \{\}
| | `sandbox` _[SandboxSettings](#sandboxsettings)_ | sandbox is where seats run code. Unset gives no seat a sandbox.
Crewlet's providers.sandbox. | | Optional: \{\}
| | `integrations` _[CompanyIntegrationSettings](#companyintegrationsettings)_ | integrations is company-wide integration behaviour. | | MinProperties: 1
Optional: \{\}
| -| `engineConfig` _[EngineConfigEntry](#engineconfigentry) array_ | engineConfig is passed through into the company document as written,
for the engine's tuning blocks that have no field here: turn_engine,
learning, scheduling, workers, skill_variables and the
notification_* settings, spelled as Crewlet spells them. Each entry is
one top-level key of the document and its value. The operator does not
interpret it; the engine validates it in the dry run before every
apply, and a refusal is reported on the Company.
It may not set what another field or object of this API owns: name,
mission, vision, policies, timezone, tracker, knowledge, token_budget,
providers, integrations, mcp_servers, roles or units. Every position
Crewlet holds a credential in lives under one of those keys, so
engineConfig can never carry a credential: the API server refuses a
Company whose engineConfig names one of them, before anything is
stored. | | MaxItems: 64
Optional: \{\}
| +| `engineConfig` _[EngineConfigEntry](#engineconfigentry) array_ | engineConfig is passed through into the company document as written,
for the engine's tuning blocks that have no field here: turn_engine,
learning, scheduling, workers, skill_variables and the
notification_* settings, spelled as Crewlet spells them. Each entry is
one top-level key of the document and its value. The operator does not
interpret it; the engine validates it in the dry run before every
apply, and a refusal is reported on the Company.
It may not set what another field or object of this API owns: name,
mission, vision, policies, timezone, tracker, knowledge, inbox,
token_budget, providers, integrations, mcp_servers, roles or units.
Every position Crewlet holds a credential in lives under one of those
keys, so engineConfig can never carry a credential: the API server
refuses a Company whose engineConfig names one of them, before anything
is stored.
Nor may it set code, Crewlet's code host, which keeps the company's git
repositories in the engine and which v1alpha1 does not offer: the host
serves them only from a node with the git role and a cache of them on
its volume, and no pod the operator runs has either, so a document
that set it would declare repositories nothing can clone or push. | | MaxItems: 64
Optional: \{\}
| | `config` _[ConfigSettings](#configsettings)_ | config is how the operator manages the company's document. | \{ managementPolicy:Enforce \} | MinProperties: 1
Optional: \{\}
| | `topology` _[CompanyTopology](#companytopology)_ | topology is how the engine is laid out in pods. | \{ data:map[replicas:1] \} | MinProperties: 1
Optional: \{\}
| | `exposure` _[CompanyExposure](#companyexposure)_ | exposure overrides how the company is reached. | | MinProperties: 1
Optional: \{\}
| @@ -752,7 +774,7 @@ _Appears in:_ | `allowedReplicas` _integer array_ | allowedReplicas is the set of data-pod counts a Company of this class
may choose in spec.topology.data.replicas. v1alpha1 runs exactly one
data pod, so the only value admitted is 1; 3 and 5, a highly available
estate, are added when the operator can run them. | [1] | MaxItems: 1
MinItems: 1
items:Enum: [1]
Optional: \{\}
| | `resources` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.37/#resourcerequirements-v1-core)_ | resources are the compute resources of the engine container. | | Optional: \{\}
| | `storage` _[DataStorage](#datastorage)_ | storage is the data volume each data pod gets. | | Required: \{\}
| -| `startupSeconds` _integer_ | startupSeconds is how long a data pod's engine may take, from its
start, to answer /health 200: the time it takes to recover what it holds
on its volume (the store, the event stream, the replicated estate) and
bring its broker up. Crewlet waits that recovery out rather than
bounding it, because it grows with the data, so the startup probe is
what bounds it: it allows at least this long from the container's
start, and less than ten seconds more. A boot that runs past it is
restarted and begins its recovery again, so err long: a window that is
too long costs nothing on a healthy boot, only the time to restart one
that hung.
It is at least every bound Crewlet puts on a data pod's boot, laid end
to end, which a window may not cut short: 1636 seconds, or 1776 with
objects.backend S3, whose bucket every boot probes. Those are its waits
for its own broker and store (provisioning the event stream, the
coordination buckets, the object store and the state logs, two minutes
a window on the broker of one member a data pod runs, and the requests
around them), never for a peer or a seat pod. One of them grows with
what the node holds and is left to the recovery's share, as the
recovery is: the claim of each batch of a seat pod's events the node
kept and could not settle, 15 s a batch. Unset derives it from the
data volume: that floor plus six seconds per GiB of the volume. That is
four times the second and a half per GiB Crewlet measured recovering
after a crash on a fast local disk, for volumes that are
network-attached; set this for disks slower than that. Nothing the
company configures counts: the engine answers /health once it has
recovered, and only then reads the company's secrets, starts its tool
servers and restores its CLIs' logins.
Changing it changes the data pods' template, which restarts them. | | Maximum: 86400
Minimum: 1636
Optional: \{\}
| +| `startupSeconds` _integer_ | startupSeconds is how long a data pod's engine may take, from its
start, to answer /health 200: the time it takes to recover what it holds
on its volume (the store, the event stream, the replicated estate) and
bring its broker up. Crewlet waits that recovery out rather than
bounding it, because it grows with the data, so the startup probe is
what bounds it: it allows at least this long from the container's
start, and less than ten seconds more. A boot that runs past it is
restarted and begins its recovery again, so err long: a window that is
too long costs nothing on a healthy boot, only the time to restart one
that hung.
It is at least every bound Crewlet puts on a data pod's boot, laid end
to end, which a window may not cut short: 1461 seconds, or 1601 with
objects.backend S3, whose bucket every boot probes. Those are its waits
for its own broker and store (provisioning the event stream, the
coordination buckets, the object store and the state logs, two minutes
a window on the broker of one member a data pod runs, and the requests
around them), never for a peer or a seat pod. One of them grows with
what the node holds and is left to the recovery's share, as the
recovery is: the claim of each batch of a seat pod's events the node
kept and could not settle, 15 s a batch. And a read of a state log's
state or record is counted at the 30 s Crewlet asks it for while the
broker answers none, past which it gives the read up, so no broker
that never answers holds a boot past the window. Unset derives it from
the data volume: that floor plus six seconds per GiB of the volume.
That is four times the second and a half per GiB Crewlet measured
recovering after a crash on a fast local disk, for volumes that are
network-attached; set this for disks slower than that. Nothing the
company configures counts: the engine answers /health once it has
recovered, and only then reads the company's secrets, starts its tool
servers and restores its CLIs' logins.
Changing it changes the data pods' template, which restarts them. | | Maximum: 86400
Minimum: 1461
Optional: \{\}
| | `terminationGracePeriodSeconds` _integer_ | terminationGracePeriodSeconds is how long a data pod may take to stop.
The engine finishes running turns before it exits and puts no bound on
that itself, so the default is five minutes, which covers a long turn,
rather than the Kubernetes default of thirty seconds, which does not. | 300 | Maximum: 3600
Minimum: 30
Optional: \{\}
| @@ -770,7 +792,7 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | | `storageClassName` _string_ | storageClassName is the StorageClass of the data volumes. Unset uses
the cluster's default class. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| -| `size` _[Quantity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.37/#quantity-resource-api)_ | size is the capacity requested for each data volume. It holds the
store, the event stream and the store's cache. There is no default: how
much a tenant may keep is a platform decision.
Three quarters of it is the engine's event stream, Crewlet's
stream.store_max_bytes: the stream reserves what a node keeps there up
front, and a fresh node refuses to boot on a share that cannot hold it
(Crewlet's engine.BrokerFloor, and its deployment guide's "minimum room
for a fresh node"). Its seven state logs' ceilings at their smallest
come to 6979321856 bytes, so a volume must be at least 9Gi, whose share
of 6.75Gi holds them. With objects.backend NATS, the default, the object
store's bucket reserves a quarter of the share beside them, 9305762474
bytes in all, so such a class must give at least 12Gi, whose share is
9Gi. That is the least that boots: the mailboxes and coordination
buckets grow into what is left, so give a company that runs for long
more. The stream's ceiling is 64Ti (config.StoreMaxBytesCeiling, a
guard against a unit mistake), so a volume larger than about 85Ti gives
the stream 64Ti and the rest to the store and its snapshots.
It applies to the data volumes of companies created after it is set.
An existing company's volumes keep the size they were created with:
they are claimed through the StatefulSet's volume claim templates,
which Kubernetes does not let change, and growing a volume is a
decision about one company's storage class and data, not the class's.
Its owner expands the company's claim instead, and once the expansion
lands the operator sizes the stream from what the claim then has. | | Required: \{\}
| +| `size` _[Quantity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.37/#quantity-resource-api)_ | size is the capacity requested for each data volume. It holds the
store, the event stream and the store's cache. There is no default: how
much a tenant may keep is a platform decision.
Three quarters of it is the engine's event stream, Crewlet's
stream.store_max_bytes: the stream reserves what a node keeps there up
front, and a fresh node refuses to boot on a share that cannot hold it
(Crewlet's engine.BrokerFloor, and its deployment guide's "minimum room
for a fresh node"). Its eight state logs' ceilings at their smallest
(the code host's log among them, which a node reserves whether or not
the company hosts code) come to 8053063680 bytes, so a volume must be
at least 10Gi, whose share of 7.5Gi holds them. With objects.backend
NATS, the default, the object store's bucket reserves a quarter of the
share beside them, 10737418239 bytes in all, so such a class must give
at least 14Gi, whose share is 10.5Gi. That is the least that boots: the
mailboxes and coordination buckets grow into what is left, so give a
company that runs for long more. The stream's ceiling is 64Ti
(config.StoreMaxBytesCeiling, a guard against a unit mistake), so a
volume larger than about 85Ti gives the stream 64Ti and the rest to the
store and its snapshots.
It applies to the data volumes of companies created after it is set.
An existing company's volumes keep the size they were created with:
they are claimed through the StatefulSet's volume claim templates,
which Kubernetes does not let change, and growing a volume is a
decision about one company's storage class and data, not the class's.
Its owner expands the company's claim instead, and once the expansion
lands the operator sizes the stream from what the claim then has.
A class stored below the floor, under an older one, keeps its size,
and every Company created under it gets a volume no fresh node boots
on, so raise it. Until it is raised the API refuses any change to the
class's spec, or, with objects.backend S3, a change of its size that
leaves it below the floor. | | Required: \{\}
| #### DataTopology @@ -1336,27 +1358,6 @@ _Appears in:_ | `sectionName` _string_ | sectionName is the name of one listener of the Gateway. Unset attaches
to every listener that accepts the route. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| -#### GitHubAccessTier - -_Underlying type:_ _string_ - -GitHubAccessTier is how much a seat may do in a repository. It maps onto -Crewlet's role integrations.github.tier (ReadOnly is `read_only`, Review is -`review`, FullAccess is `full_access`). - -_Validation:_ -- Enum: [ReadOnly Review FullAccess] - -_Appears in:_ -- [GitHubSeatIntegration](#githubseatintegration) - -| Field | Description | -| --- | --- | -| `ReadOnly` | | -| `Review` | | -| `FullAccess` | | - - #### GitHubApp @@ -1427,7 +1428,8 @@ _Appears in:_ GitHubSeatIntegration is a seat's identity on GitHub. It maps onto Crewlet's role integrations.github and needs the company's GitHub -Integration. +Integration. How much the identity may do is the seat's spec.code.access, +which its app is created with and mints every token at. _Validation:_ - MinProperties: 1 @@ -1437,7 +1439,6 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `tier` _[GitHubAccessTier](#githubaccesstier)_ | tier is how much the seat may do in a repository. Unset is ReadOnly.
Crewlet's `tier`. | | Enum: [ReadOnly Review FullAccess]
Optional: \{\}
| | `repos` _string array_ | repos are the repositories the seat works in, as owner/name. Unset is
every repository the app's installation covers. Crewlet's `repos`. | | MaxItems: 64
items:MaxLength: 201
items:Pattern: `^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$`
Optional: \{\}
| | `app` _[GitHubApp](#githubapp)_ | app is the GitHub App the seat acts as. | | Optional: \{\}
| @@ -1570,6 +1571,30 @@ _Appears in:_ | `accessLevel` _[GitLabAccessLevel](#gitlabaccesslevel)_ | accessLevel is the seat's membership level. | | Enum: [Developer Maintainer]
Required: \{\}
| +#### InboxSettings + + + +InboxSettings is the company's policy over a person's inbox: the notices +the engine writes to each person its work concerns. It maps onto +Crewlet's inbox block. + +It is the company's rather than its tracker's, and applies whatever +tracker the company runs: a person's inbox holds the notices of every part +of the engine that writes them, Worklet's among them, and one horizon ages +them all. + +_Validation:_ +- MinProperties: 1 + +_Appears in:_ +- [CompanySpec](#companyspec) + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `retentionSeconds` _integer_ | retentionSeconds is how long a person's inbox keeps a notice; the
history the notice points at is kept regardless. Crewlet keeps the
horizon in whole days, so it is a whole number of days in seconds,
from 30 days (2592000) to 3650 days (315360000), such as 7776000 for
90 days. Unset takes Crewlet's default of 365 days. Crewlet's
`inbox.retention_days`, which is this divided by 86400. | | Maximum: 3.1536e+08
Minimum: 2.592e+06
Optional: \{\}
| + + #### IngressExposure @@ -2663,6 +2688,26 @@ _Appears in:_ | `pathStyle` _boolean_ | pathStyle addresses the bucket in the URL path rather than in the host
name; MinIO and most self-hosted gateways need it. | | Optional: \{\}
| +#### SandboxEnvVar + + + +SandboxEnvVar is a variable a sandbox setup step merges into the +environment of its commands and of a coding run. Crewlet treats every value +there as a credential, so a literal value is not accepted: the value always +comes from a Secret. + + + +_Appears in:_ +- [SandboxSetupStep](#sandboxsetupstep) + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `name` _string_ | name is the variable's name. It may not be one Crewlet sets in a coding
run itself, which Crewlet refuses in every company document:
CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT,
CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's
ReservedSandboxEnv). | | MaxLength: 128
MinLength: 1
Pattern: `^[A-Za-z_][A-Za-z0-9_]*$`
Required: \{\}
| +| `valueFrom` _[SecretValueSource](#secretvaluesource)_ | valueFrom is the Secret key the value is read from. | | Required: \{\}
| + + #### SandboxFile @@ -2730,7 +2775,7 @@ _Appears in:_ | `name` _string_ | name identifies the step in logs and failure messages, so Crewlet
refuses one of white space alone, and so does the API. | | MaxLength: 63
MinLength: 1
Required: \{\}
| | `files` _[SandboxFile](#sandboxfile) array_ | files are written into the sandbox before the commands run. Crewlet's
`files`, a credential position. | | MaxItems: 16
Optional: \{\}
| | `commands` _string array_ | commands are shell commands run in order; one that exits non-zero
fails the sandbox. Crewlet's `commands`. Crewlet resolves nothing in a
command: the sandbox's login shell expands $NAME and $\{NAME\} from the
box's own environment (its HOME and PATH among it) and the run's,
never from the engine's. The run's is the steps' env, the Seat's
sandbox env, and what Crewlet adds: the agent's handle and, when it
has one, its email, and the variables that sign the coding agent in
to the provider its code work runs on, or, in an agent-mode
executor's run, those of the executor's own provider
(docs/guides/credentials.md says which, and when the seat's own env
leaves them out). The API refuses $\{NAME\} (NAME an identifier) all
the same, as the operator refuses every $\{NAME\} it did not write in a
string it renders: write a sandbox variable as $NAME. Parameter
expansions such as $\{1:-x\} are allowed. | | MaxItems: 32
items:MaxLength: 4096
items:MinLength: 1
Optional: \{\}
| -| `env` _[SecretEnvVar](#secretenvvar) array_ | env is merged into the environment of the commands and of the run.
Crewlet's `env`, a credential position. | | MaxItems: 32
Optional: \{\}
| +| `env` _[SandboxEnvVar](#sandboxenvvar) array_ | env is merged into the environment of the commands and of the run.
It may not set a variable Crewlet sets in a coding run itself, which
Crewlet refuses in every company document, as a Seat's
spec.sandbox.env may not: CREWLET_GIT_TOKEN, CREWLET_GIT_URL,
CREWLET_GIT_TRAILER_SEAT, CREWLET_GIT_TRAILER_RUN and
CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's ReservedSandboxEnv). Crewlet's
`env`, a credential position. | | MaxItems: 32
Optional: \{\}
| | `optionalKeys` _string array_ | optionalKeys names variables of env, and paths of files, whose value may go
missing. Crewlet refuses to start a run_sandbox coding run whose environment
or files hold a value that does not resolve: a Secret that does not exist,
holds no such key or holds an empty value. Under an entry listed here, such
a variable is left out of the run and such a file is not written, and the
run starts without it. A missing Secret or key under one is then no failure
of the object's references and holds nothing back. Under the Enforce policy
the operator also takes the value it pushed for it out of the engine's
secret store, and has the engine read the store again whether or not the
document can be applied, so no run starts with a deleted Secret's value, but
in a revision the engine runs that reads the key as required (one from
before the key was listed) or whose reading the operator does not know
(another document somebody else activated), which keeps the value until the
next document runs, and that one is activated once with it and at once again
without it; a Secret key that is also read where it is required holds the
document while it is missing, and keeps the value it had in a revision the
engine runs that requires it too until the document applies. A later step's
variable of the same name wins over this one's, and the Seat's
spec.sandbox.env over every step's, and the list beside the value that wins
is the one that counts for it; a file is the step's own. Crewlet's
`optional_keys`. | | MaxItems: 48
items:MaxLength: 1024
items:MinLength: 1
Optional: \{\}
| | `brief` _string_ | brief tells the coding agent what this step made true about the
sandbox. | | MaxLength: 2048
MinLength: 1
Optional: \{\}
| | `timeoutSeconds` _integer_ | timeoutSeconds caps each command of the step. | | Minimum: 1
Optional: \{\}
| @@ -2875,6 +2920,31 @@ a letter or digit, at most 63 characters. The handle cannot change. | `status` _[SeatStatus](#seatstatus)_ | status is the seat's observed state. | | MinProperties: 1
Optional: \{\}
| +#### SeatCode + + + +SeatCode is an agent seat's access to the company's code. It maps onto +Crewlet's role `code`, which holds one tier for every code host the engine +drives: the seat's GitHub App is created with, and mints every token at, +the permissions of its tier, and Crewlet's own code host, which this API +does not offer, caps what the org chart grants the seat by the same tier. +The tier is a setting of the seat's rather than of its GitHub identity, so +it says the same on every host. A person acts as their own identity on +every host, so a tier on a Human seat would cap nothing, and the API +refuses one. + +_Validation:_ +- MinProperties: 1 + +_Appears in:_ +- [SeatSpec](#seatspec) + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `access` _[CodeAccessTier](#codeaccesstier)_ | access is how much of a repository the seat may do. Unset is
ReadOnly, the least the engine hands out. Crewlet's `access`. The
engine warns that the tier grants nothing where nothing applies it:
in a company with no GitHub Integration, or for a seat without
spec.integrations.github. | | Enum: [ReadOnly Review FullAccess]
Optional: \{\}
| + + #### SeatContact @@ -2931,7 +3001,8 @@ _Appears in:_ SeatEnvVar is a named credential of a seat delivered as an environment -variable: from a Secret, or provisioned by the engine. +variable of its sandbox runs (a Seat's spec.sandbox.env): from a Secret, or +provisioned by the engine. @@ -2940,7 +3011,7 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `name` _string_ | name is the variable's name. | | MaxLength: 128
MinLength: 1
Pattern: `^[A-Za-z_][A-Za-z0-9_]*$`
Required: \{\}
| +| `name` _string_ | name is the variable's name. It may not be one Crewlet sets in a coding
run itself, which Crewlet refuses in every company document:
CREWLET_GIT_TOKEN, CREWLET_GIT_URL, CREWLET_GIT_TRAILER_SEAT,
CREWLET_GIT_TRAILER_RUN or CREWLET_GIT_TRAILER_WORK_ITEM (Crewlet's
ReservedSandboxEnv). | | MaxLength: 128
MinLength: 1
Pattern: `^[A-Za-z_][A-Za-z0-9_]*$`
Required: \{\}
| | `valueFrom` _[SeatCredentialSource](#seatcredentialsource)_ | valueFrom is where the value comes from. | | MinProperties: 1
Required: \{\}
| @@ -3046,7 +3117,7 @@ _Appears in:_ | `pausedBy` _string_ | pausedBy is who paused the seat, as Crewlet records the author of a
write: the seat of the person who paused it, or token: for a
token no seat is bound to. A pause the operator took reads
token:operator. | | MaxLength: 256
MinLength: 1
Optional: \{\}
| | `pausedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.37/#time-v1-meta)_ | pausedAt is when the seat was paused, to the second. | | Optional: \{\}
| | `reason` _string_ | reason is why, in the pauser's words. | | MaxLength: 500
MinLength: 1
Optional: \{\}
| -| `stopRunning` _boolean_ | stopRunning is whether the pause also ended the turn the seat was on. | | Optional: \{\}
| +| `stopRunning` _boolean_ | stopRunning is whether the pause also ended the turn the seat was on,
and refuses its detached coding runs' writes until it is resumed. | | Optional: \{\}
| #### SeatPlacement @@ -3243,7 +3314,7 @@ _Appears in:_ | `pauseTTLSeconds` _integer_ | pauseTTLSeconds is how long the seat's sandbox stays paused while it
waits on a person. Unset takes the Company's default; 0 never pauses.
Crewlet's `pause_ttl_seconds`. | | Minimum: 0
Optional: \{\}
| | `maxTurns` _integer_ | maxTurns caps the agentic rounds of the seat's coding runs. Unset
takes the Company's default; 0 is uncapped. Crewlet's `max_turns`. | | Minimum: 0
Optional: \{\}
| | `mcpServerRefs` _[LocalObjectReference](#localobjectreference) array_ | mcpServerRefs name the MCPServers the coding agent may use. Crewlet's
mcp.servers. | | MaxItems: 16
Optional: \{\}
| -| `env` _[SeatEnvVar](#seatenvvar) array_ | env is the environment of the seat's sandbox runs, for the external
tokens its code work needs: from Secrets, or provisioned by the
engine, such as the GitLab token a provisioning Integration mints for
the seat, named as its tool server's is so both are one token. A
ClaudeCode sign-in here (CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY,
ANTHROPIC_AUTH_TOKEN or a CLAUDE_CODE_USE_* toggle) keeps the key and
endpoint of the provider the code work runs on out of the box, unless
a run goes without its value (optionalKeys), when the box is given
them as it is with no sign-in; and a variable named as one the
provider would set wins over it. Crewlet's `env`, a credential
position. | | MaxItems: 32
Optional: \{\}
| +| `env` _[SeatEnvVar](#seatenvvar) array_ | env is the environment of the seat's sandbox runs, for the external
tokens its code work needs: from Secrets, or provisioned by the
engine, such as the GitLab token a provisioning Integration mints for
the seat, named as its tool server's is so both are one token. A
ClaudeCode sign-in here (CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY,
ANTHROPIC_AUTH_TOKEN or a CLAUDE_CODE_USE_* toggle) keeps the key and
endpoint of the provider the code work runs on out of the box, unless
a run goes without its value (optionalKeys), when the box is given
them as it is with no sign-in; and a variable named as one the
provider would set wins over it. It may not set a variable Crewlet
sets in a coding run itself, which Crewlet refuses in every company
document, and nor may a setup step's env: CREWLET_GIT_TOKEN and
CREWLET_GIT_URL, a run's credential on the company's own code host and
the base its clones and pushes reach, and CREWLET_GIT_TRAILER_SEAT,
CREWLET_GIT_TRAILER_RUN and CREWLET_GIT_TRAILER_WORK_ITEM, the values
its commit hook writes into every commit's trailers (Crewlet's
ReservedSandboxEnv). No run of a company the operator runs is given
them, since the operator does not offer the code host, but the names
are the engine's all the same. Crewlet's `env`, a credential position. | | MaxItems: 32
Optional: \{\}
| | `optionalKeys` _string array_ | optionalKeys names variables of env whose value may go missing. Crewlet
refuses to start a run_sandbox coding run whose environment holds a value
that does not resolve: a Secret that does not exist, holds no such key or
holds an empty value, or a provisioned value nothing mints. Under a variable
listed here, such a value is left out of the run instead, which starts
without it. A missing Secret or key under one is then no failure of the
Seat's references and holds nothing back. Under the Enforce policy the
operator also takes the value it pushed for it out of the engine's secret
store, and has the engine read the store again whether or not the document
can be applied, so no run starts with a deleted Secret's value, but in a
revision the engine runs that reads the key as required (one from before the
key was listed) or whose reading the operator does not know (another
document somebody else activated), which keeps the value until the next
document runs, and that one is activated once with it and at once again
without it; a Secret key that is also read where it is required holds the
document while it is missing, and keeps the value it had in a revision the
engine runs that requires it too until the document applies. A provisioned
value an Integration mints for the seat is waited for whatever this says.
env wins over a setup step's variable of the same name, and so does this
list over the step's optionalKeys for it. Crewlet's `optional_keys`. | | MaxItems: 32
items:MaxLength: 128
items:MinLength: 1
items:Pattern: `^[A-Za-z_][A-Za-z0-9_]*$`
Optional: \{\}
| | `setup` _[SandboxSetupStep](#sandboxsetupstep) array_ | setup is the seat's own provisioning, applied after the Company's.
Crewlet's `setup`. | | MaxItems: 16
Optional: \{\}
| @@ -3282,7 +3353,8 @@ _Appears in:_ | `placement` _[SeatPlacement](#seatplacement)_ | placement is where the seat runs. Unset runs it in the Company's
spec.topology.defaultSeatPool. | | Optional: \{\}
| | `paused` _boolean_ | paused says whether the seat may take work. True pauses it: it starts
no new turn, the mail it is sent waits on its inbox in order, and its
schedules' fires are skipped rather than queued; the turn it is on
finishes first unless stopRunning is set. False resumes it, and what
waited is delivered first. Unset leaves the seat's pause to the people
who run the company in the engine's dashboard: the operator reports a
pause they take, and never lifts it.
A pause is not part of the company document. Crewlet keeps it in its
coordination store, changed through its operator catalogue's
pause_seat and resume_seat tools, which the operator calls under the
Company's Enforce policy whenever the seat's pause differs from this
field; under Observe and Seed it only reports the difference. | | Optional: \{\}
| | `pauseReason` _string_ | pauseReason is why the seat is paused, shown on the seat and in its
feed in the engine. Only with paused true. The engine keeps the
reason a pause was taken with, so a reason changed while the seat
stays paused shows from the seat's next pause on. Crewlet's pause
`reason`. | | MaxLength: 500
MinLength: 1
Optional: \{\}
| -| `stopRunning` _boolean_ | stopRunning ends the turn the seat is on at its next round when it is
paused, rather than letting the turn finish; the stopped turn is not
run again. Only with paused true. Setting it on a seat already paused
stops the turn it is on then; unsetting it changes nothing, since a
stopped turn stays stopped. Crewlet's pause `stop_running`. | | Optional: \{\}
| +| `stopRunning` _boolean_ | stopRunning ends the turn the seat is on at its next round when it is
paused, rather than letting the turn finish; the stopped turn is not
run again. A coding run the seat detached runs on, but every change
it would make is refused until the seat is resumed: a call it makes
through the tool bridge of any of the seat's tools not marked
read-only, and a push its box makes to the company's code host, which
the operator does not offer. Only with paused true. Setting it on a
seat already paused stops the turn it is on then, and the writes of
its runs; unsetting it changes nothing, since a stop lasts until the
seat is resumed. Crewlet's pause `stop_running`. | | Optional: \{\}
| +| `code` _[SeatCode](#seatcode)_ | code is the seat's access to the company's code, on every code host
the engine drives. Unset runs the seat at access ReadOnly. Crewlet's
`code`. | | MinProperties: 1
Optional: \{\}
| | `integrations` _[SeatIntegrations](#seatintegrations)_ | integrations is the seat's own identity on the company's
integrations. | | MinProperties: 1
Optional: \{\}
| | `project` _string_ | project is the tracker project a seat outside every unit owns, by its
key; a seat in a unit takes its unit's. Crewlet requires a container
key to be an upper-case letter followed by 1 to 9 upper-case letters or
digits, such as LEAD. Crewlet's `project`. | | MaxLength: 10
MinLength: 2
Pattern: `^[A-Z][A-Z0-9]\{1,9\}$`
Optional: \{\}
| | `space` _string_ | space is the knowledge container a seat outside every unit owns, by
its key, which follows the same rule as project; a seat in a unit
takes its unit's. Crewlet's `space`. | | MaxLength: 10
MinLength: 2
Pattern: `^[A-Z][A-Z0-9]\{1,9\}$`
Optional: \{\}
| @@ -3338,15 +3410,14 @@ _Appears in:_ SecretEnvVar is a named credential delivered as an environment variable, -such as a sandbox setup step's token or a coding CLI's model key. Crewlet -treats every value in the positions that use this type as a credential, so -a literal value is not accepted: the value always comes from a Secret. +such as a coding CLI's model key. Crewlet treats every value in the +positions that use this type as a credential, so a literal value is not +accepted: the value always comes from a Secret. _Appears in:_ - [LLMProviderSpec](#llmproviderspec) -- [SandboxSetupStep](#sandboxsetupstep) | Field | Description | Default | Validation | | --- | --- | --- | --- | @@ -3422,6 +3493,7 @@ _Appears in:_ - [EnvVar](#envvar) - [MCPEnvVar](#mcpenvvar) - [MCPHeader](#mcpheader) +- [SandboxEnvVar](#sandboxenvvar) - [SandboxFile](#sandboxfile) - [SecretEnvVar](#secretenvvar) @@ -3670,7 +3742,6 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | | `backend` _[TrackerBackend](#trackerbackend)_ | backend is which tracker the company runs. | Worklet | Enum: [Worklet Jira None]
Optional: \{\}
| -| `worklet` _[WorkletSettings](#workletsettings)_ | worklet is the policy of Worklet, for backend Worklet. | | MinProperties: 1
Optional: \{\}
| #### Unit @@ -3797,7 +3868,7 @@ _Appears in:_ | `gateway` _[GatewayExposure](#gatewayexposure)_ | gateway is the Gateways to attach to, for mode Gateway. | | Optional: \{\}
| | `ingress` _[IngressExposure](#ingressexposure)_ | ingress is the Ingress settings, for mode Ingress. | | Optional: \{\}
| | `external` _[ExternalExposure](#externalexposure)_ | external is where the forwarder runs, for mode External. | | MinProperties: 1
Optional: \{\}
| -| `hostname` _string_ | hostname is the public host name of each company, as a template:
\{\{namespace\}\}, \{\{name\}\} and \{\{uid\}\} are replaced by the Company's
namespace, name and UID. The result is the company's public base URL,
https://, which every vendor is told to deliver to. Prefer
\{\{uid\}\}: the host name is public, and a UID says nothing about the
tenant where a name can; and a UID is never reused, so a vendor still
configured for a deleted company can never reach a company created
later under the same name. The template must contain \{\{uid\}\}, or
\{\{name\}\} and \{\{namespace\}\} joined by a '.', which neither can hold, so
that no two companies share a host name: with a '-' between them,
team-a/b-web and team-a-b/web would. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| +| `hostname` _string_ | hostname is the public host name of each company, as a template:
\{\{namespace\}\}, \{\{name\}\} and \{\{uid\}\} are replaced by the Company's
namespace, name and UID. The result is the company's public base URL,
https://, which every vendor is told to deliver to. Prefer
\{\{uid\}\}: the host name is public, and a UID says nothing about the
tenant where a name can; and a UID is never reused, so a vendor still
configured for a deleted company can never reach a company created
later under the same name. The template must contain \{\{uid\}\}, or
\{\{name\}\} and \{\{namespace\}\} joined by a '.', which neither can hold, so
that no two companies share a host name: with a '-' between them,
team-a/b-web and team-a-b/web would.
The engine hands the address to its sandboxes too, as its tool
bridge's and its telemetry receiver's, and refuses to start on one the
URL parser of a sandbox's coding agent reads as an IPv4 address that
is none, as it does a host whose last label is a number. So that last
label may not be one (\{\{uid\}\}.123, \{\{uid\}\}.0x1f). One a variable fills
is a number only for some Companies, as api.hostname says, and the
operator holds that Company's engine back, with RuntimeReady=False
AddressRefused, while it hands the address to a sandbox. | | MaxLength: 253
MinLength: 1
Optional: \{\}
| #### WebhookExposureMode @@ -3841,21 +3912,3 @@ _Appears in:_ | `Addressed` | WorkingStatusAddressed shows it only when a person is plausibly
waiting: a direct message, a mention, or a thread the seat follows.
| -#### WorkletSettings - - - -WorkletSettings is the policy of Worklet. It maps onto Crewlet's -tracker.worklet. - -_Validation:_ -- MinProperties: 1 - -_Appears in:_ -- [TrackerSettings](#trackersettings) - -| Field | Description | Default | Validation | -| --- | --- | --- | --- | -| `inboxRetentionDays` _integer_ | inboxRetentionDays is how long a person's inbox keeps a row; the
history the row points at is kept regardless. Unset takes Crewlet's
default of 365. Crewlet's `inbox_retention_days`. | | Maximum: 3650
Minimum: 30
Optional: \{\}
| - - diff --git a/internal/controller/company_exposure.go b/internal/controller/company_exposure.go index 9dcefc1..57bd0ef 100644 --- a/internal/controller/company_exposure.go +++ b/internal/controller/company_exposure.go @@ -373,24 +373,37 @@ func engineAddressesOf(c *crewletaiv1alpha1.Company, routed, artifactsRouted str } // addressRefusalMessage is RuntimeReady's message when Crewlet would refuse to -// start on an address engineAddressesOf chose: which it is, where it came -// from and what to change. The API refuses the addresses a Company states -// that Crewlet would refuse, so the one that reaches here is in practice the -// API route's host, which the class's template makes of the Company's -// namespace and name. A Company's own address reaches here only when the -// API admitted it before it had the rule that refuses it, or on an API -// server whose URL parser takes a shape Crewlet's refuses that no rule -// covers. +// start on an address engineAddressesOf chose, or one boxURLs hands the +// sandboxes: which it is, where it came from and what to change. The API +// refuses the addresses a Company states that Crewlet would refuse, so the +// one that reaches here is in practice a route's host, which the class's +// template makes of the Company's namespace and name: the API route's, or the +// webhooks route's where the engine hands it to sandboxes. A Company's own +// address reaches here only when the API admitted it before it had the rule +// that refuses it, or on an API server whose URL parser takes a shape +// Crewlet's refuses that no rule covers. func addressRefusalMessage(c *crewletaiv1alpha1.Company, class *crewletaiv1alpha1.EngineClass, refused *render.AddressError) string { // Where the address came from, as engineAddressesOf chose it. The // addresses the operator chooses for an engine nothing routes to // (localhost, the public listener's in-cluster name) are ones Crewlet - // takes, so they need no sentence; and so is the webhooks route's, - // https:// and a host name the class's rules hold to a DNS name, with no - // port, query or fragment, which is all Crewlet asks of - // api.public.external_url. + // takes, so they need no sentence; and so is the webhooks route's as + // api.public.external_url, https:// and a host name the class's rules + // hold to a DNS name, with no port, query or fragment, which is all + // Crewlet asks of that key. Handed to sandboxes, it is held to their URL + // parser too, which a host whose last label is a number fails. + box := refused.Field == render.EnvBridgeURL || refused.Field == render.EnvReceiverURL var where string switch { + case box && c.Spec.Exposure.PublicBaseURL != "": + where = " It is the Company's spec.exposure.publicBaseURL, which the engine hands its sandboxes: correct it." + case box && class.Spec.Exposure.Webhooks.Hostname != "": + where = fmt.Sprintf(" It is the address of the webhooks route of EngineClass %s, its spec.exposure.webhooks.hostname "+ + "%q expanded for this Company, which the engine hands its sandboxes. A template whose last label holds "+ + "{{namespace}} or {{name}} ends in a number for some Companies: one whose namespace is a number (4711, or 0x "+ + "and hex digits such as 0x1f), or whose namespace or name makes one with the template's own characters beside "+ + "it (0x{{name}} for a Company named cafe). End the template in a domain of your own, such as "+ + "{{uid}}.hooks.example.com, or run the Company in a namespace of another name.", + class.Name, class.Spec.Exposure.Webhooks.Hostname) case refused.Field == render.FieldArtifactsURL && c.Spec.Exposure.ArtifactsBaseURL != "": where = " It is the Company's spec.exposure.artifactsBaseURL: correct it, giving artifacts a host of their own." case refused.Field == render.FieldArtifactsURL && class.Spec.Exposure.Artifacts.Hostname != "": @@ -401,7 +414,7 @@ func addressRefusalMessage(c *crewletaiv1alpha1.Company, class *crewletaiv1alpha where = " It is the Company's spec.exposure.publicBaseURL: correct it." case refused.Field == render.FieldExternalURL && c.Spec.Exposure.DashboardBaseURL != "": where = " It is the Company's spec.exposure.dashboardBaseURL: correct it." - case refused.Field != render.FieldPublicURL && class.Spec.Exposure.API.Hostname != "": + case !box && refused.Field != render.FieldPublicURL && class.Spec.Exposure.API.Hostname != "": where = fmt.Sprintf(" It is the address of the API route of EngineClass %s, its spec.exposure.api.hostname %q "+ "expanded for this Company. A template whose last label holds {{namespace}} or {{name}} ends in a number "+ "for some Companies: one whose namespace is a number (4711, or 0x and hex digits such as 0x1f), or whose "+ @@ -763,12 +776,13 @@ func engineEgress(settings *crewletaiv1alpha1.NetworkPolicySettings) []*networki // that closes the listener reaches the pods only as the data StatefulSet // rolls, which takes a drain at least and lasts as long as the rollout is // held back, and a pod that still listens meanwhile must stay behind the -// policy. Crewlet's broker accepts any connection on -// that port, with no authentication and no TLS, and whoever joins it reads -// and writes the company's whole event stream; with companies sharing a -// cluster, and sometimes a namespace, only a policy keeps another's pods off -// it. It admits the leaf port from this company's seat pods alone, selected -// by the company's label. +// policy. Crewlet's broker admits a link on that port only on a key derived +// from the company's keyring, but it encrypts nothing, and whoever joins it +// reads and writes the company's whole event stream; with companies sharing a +// cluster, and sometimes a namespace, the policy keeps another's pods from +// reaching the port at all, so the keyring is not the only thing between +// them and the stream. It admits the leaf port from this company's seat pods +// alone, selected by the company's label. // // It selects the data pods for ingress, and a pod any policy selects for // ingress is denied every connection no policy admits: so without the class's diff --git a/internal/controller/company_exposure_test.go b/internal/controller/company_exposure_test.go index 9cd87b3..d677155 100644 --- a/internal/controller/company_exposure_test.go +++ b/internal/controller/company_exposure_test.go @@ -119,10 +119,12 @@ const ( // TestAddressRefusalNamesWhereTheAddressCameFrom pins RuntimeReady's message // for an address Crewlet would refuse to start on: it names the Tier A key, -// the address and the engine's reason, and sends the reader to where -// engineAddressesOf took it from, the Company's own field or the class's API -// host name template; and names no source for an address the operator chose -// itself for an engine nothing routes to, nor for the webhooks route's. +// or the variable that hands the address to sandboxes, the address and the +// engine's reason, and sends the reader to where engineAddressesOf took it +// from, the Company's own field or the class's API host name template, or, +// for an address handed to sandboxes, the webhooks host name template; and +// names no source for an address the operator chose itself for an engine +// nothing routes to, nor for the webhooks route's as api.public.external_url. func TestAddressRefusalNamesWhereTheAddressCameFrom(t *testing.T) { const reason = "has a host a browser reads as an IPv4 address" for _, c := range []struct { @@ -165,6 +167,22 @@ func TestAddressRefusalNamesWhereTheAddressCameFrom(t *testing.T) { name: "an address the operator chose", field: render.FieldExternalURL, wantNot: []string{"It is"}, }, + { + name: "the Company's public address, handed to the tool bridge's sandboxes", field: render.EnvBridgeURL, + public: "https://hooks.example.123", apiHost: routeOfNamespace, webhooksHost: webhooksHostname, + want: []string{"spec.exposure.publicBaseURL, which the engine hands its sandboxes: correct it"}, + wantNot: []string{hostnameWord}, + }, + { + // Never sent to the API route, though the class names one. + name: "the webhooks route's address, handed to the telemetry receiver's sandboxes", field: render.EnvReceiverURL, + apiHost: routeOfNamespace, webhooksHost: "{{name}}.{{namespace}}", + want: []string{ + `the webhooks route of EngineClass standard, its spec.exposure.webhooks.hostname "{{name}}.{{namespace}}"`, + "which the engine hands its sandboxes", + }, + wantNot: []string{"api.hostname"}, + }, { name: "the Company's artifact origin", field: render.FieldArtifactsURL, artifacts: "https://crewlet.example.com", apiHost: routeOfNamespace, artifactsHost: "{{uid}}.usercontent.example", diff --git a/internal/controller/company_pools.go b/internal/controller/company_pools.go index a66573c..93ff2c3 100644 --- a/internal/controller/company_pools.go +++ b/internal/controller/company_pools.go @@ -83,11 +83,16 @@ const ( // - 60: a second connection, for coordination (listenTimeout twice); // - 300: the coordination buckets, a window of their own // (clusterSequenceBudget); - // - 90: the fleet's record of its object store (AgreeObjectBackend): - // eight rounds of a write and a read at the client's 5 s default and - // a read after them, begun again within 5 s; a boot meets more than - // three of those requests only if something removes the record - // while it runs, but the rounds are the engine's bound; + // - 85: the fleet's record of its object store (AgreeObjectBackend): + // one attempt, a create of at most eight rounds of a write and a + // read at the client's 5 s default (createKey) and a read after it. + // The engine bounds the agreement by attempts, not time (casRetry, + // up to fleetCASAttempts of them, each after a jittered wait that + // grows with every attempt lost), which no window can lay end to end. + // It makes another only when the record is gone between the + // create's refusal and the read, which only someone removing the + // record does, so a boot nobody does that to meets one, and one is + // what is counted; // - 61: the lock on the store's native library, which a container // killed while extracting it leaves on the scratch volume a restart // keeps (tursoPrepareTimeout, a minute, polled every 25 ms); @@ -97,7 +102,7 @@ const ( // below. Nothing on a seat pod's boot waits on what the company // configures or grows with what it holds: the engine lists the // company's secrets after /health answers 200. - seatPodBootSeconds = 60 + 61 + 300 + 60 + 300 + 90 + 61 + 20 + seatPodBootSeconds = 60 + 61 + 300 + 60 + 300 + 85 + 61 + 20 // seatPodNATSObjectsSeconds is the object store's step on the default // backend: the broker's limit read to size the bucket (LookupBudget, // 30 s), then the bucket looked up (30 s), created (2 min, Budget), diff --git a/internal/controller/company_pools_test.go b/internal/controller/company_pools_test.go index 3ae1088..02e58a9 100644 --- a/internal/controller/company_pools_test.go +++ b/internal/controller/company_pools_test.go @@ -419,7 +419,7 @@ var _ = Describe("Company seat pools", func() { "liveness restarts the engine before its watchdog's 45 seconds") Expect(container.ReadinessProbe.HTTPGet.Path).To(Equal("/ready"), "ready once the pod has joined and may claim seats") Expect(container.StartupProbe.HTTPGet.Path).To(Equal("/health")) - Expect((container.StartupProbe.FailureThreshold-1)*container.StartupProbe.PeriodSeconds).To(BeNumerically(">=", 1142), + Expect((container.StartupProbe.FailureThreshold-1)*container.StartupProbe.PeriodSeconds).To(BeNumerically(">=", 1137), "every bound the engine puts on a leaf's boot, with its object store on the event stream") Expect(*container.SecurityContext.ReadOnlyRootFilesystem).To(BeTrue()) Expect(container.Env[:len(want.Env)]).To(Equal(want.Env)) diff --git a/internal/controller/company_runtime.go b/internal/controller/company_runtime.go index b3939ad..ea17bb9 100644 --- a/internal/controller/company_runtime.go +++ b/internal/controller/company_runtime.go @@ -234,6 +234,7 @@ func (r *CompanyReconciler) runWorkload( PublicURL: in.addresses.public, AllowedOrigins: in.addresses.origins, ArtifactsURL: in.addresses.artifacts, + BoxURLs: boxURLs(c, class, in.bridgeURL), }) if errors.Is(err, render.ErrNoAddress) { // engineAddressesOf always names both, so this is the operator's diff --git a/internal/controller/company_runtime_test.go b/internal/controller/company_runtime_test.go index 9db8042..25febef 100644 --- a/internal/controller/company_runtime_test.go +++ b/internal/controller/company_runtime_test.go @@ -431,7 +431,7 @@ var _ = Describe("Company runtime", func() { Expect(container.StartupProbe.HTTPGet.Path).To(Equal("/health")) Expect(container.StartupProbe.PeriodSeconds).To(Equal(int32(probePeriodSeconds))) Expect((container.StartupProbe.FailureThreshold-1)*container.StartupProbe.PeriodSeconds). - To(BeNumerically(">=", 1636+16*6), + To(BeNumerically(">=", 1461+16*6), "the startup probe gives the boot every bound the engine puts on it and six seconds for every GiB of the "+ "class's 16Gi volume, counted from a first probe that may land at once") Expect(container.StartupProbe.TimeoutSeconds).To(Equal(int32(5)), @@ -1095,7 +1095,7 @@ var _ = Describe("Company runtime", func() { NotTo(Equal(before.Spec.Template.Annotations[BootstrapHashAnnotation])) Expect(after.Spec.VolumeClaimTemplates[0].Spec.Resources.Requests.Storage().String()).To(Equal("16Gi"), "the template stays as it was made") - Expect(engine(after).StartupProbe.FailureThreshold).To(Equal(startupFailureThreshold(1636+32*6)), + Expect(engine(after).StartupProbe.FailureThreshold).To(Equal(startupFailureThreshold(1461+32*6)), "the startup window covers the recovery of the volume the pod has") }) diff --git a/internal/controller/company_workload.go b/internal/controller/company_workload.go index a1c6be1..fcc9e8c 100644 --- a/internal/controller/company_workload.go +++ b/internal/controller/company_workload.go @@ -76,8 +76,8 @@ const BootstrapHashAnnotation = "crewlet.ai/bootstrap-hash" // LeafListenerAnnotation marks a data pod whose engine opens the leaf // listener (render.LeafPort), with the value leafListenerOpen. It is how the // operator knows what the pods that run open, rather than what their next -// template will: the listener accepts any connection, with no -// authentication, so the NetworkPolicy that guards it (-broker) stays +// template will: the listener admits whoever holds the company's keyring, +// and encrypts nothing, so the NetworkPolicy that guards it (-broker) stays // until no running data pod opens it, however long a rollout that closes it // takes or is held back. const LeafListenerAnnotation = "crewlet.ai/leaf-listener" @@ -94,8 +94,8 @@ const ( envOTLPProtocol = "OTEL_EXPORTER_OTLP_PROTOCOL" envOTLPHeaders = "OTEL_EXPORTER_OTLP_HEADERS" envOTELResourceAtt = "OTEL_RESOURCE_ATTRIBUTES" - envMCPBridgeURL = "CREWLET_MCP_BRIDGE_URL" - envOTelReceiverURL = "CREWLET_SANDBOX_OTEL_RECEIVER_URL" + envMCPBridgeURL = render.EnvBridgeURL + envOTelReceiverURL = render.EnvReceiverURL ) // preStopSleepSeconds is how long a stopping data pod keeps serving before @@ -129,7 +129,7 @@ const probePeriodSeconds = 5 // period short of threshold periods. So the threshold covers the window with // threshold-1 periods, rounded up. A threshold of the window over the period // truncated, as it was, cut a window that is no multiple of the period (a -// class's startupSeconds of 1637, the 1732 seconds a 16Gi volume derives) by +// class's startupSeconds of 1462, the 1557 seconds a 16Gi volume derives) by // up to four seconds more on top of that period. func startupFailureThreshold(seconds int32) int32 { return (seconds+probePeriodSeconds-1)/probePeriodSeconds + 1 @@ -155,36 +155,51 @@ const ( // - 120: the event stream's own streams (soloSequenceBudget); // - 60: a second connection, for coordination; // - 120: the coordination buckets, a window of their own; - // - 90: the fleet's record of its object store, as on a seat pod + // - 85: the fleet's record of its object store, as on a seat pod // (seatPodBootSeconds); // - 40: four registrations of what the node answers (flushBudget); - // - 40: its admission, a write and seven reads at 5 s (admit); - // - 10: the estate service's registration; + // - 45: its admission, a write and a read of each log's maintenance + // window, eight reads, at 5 s each (admit); + // - 20: the estate service's registrations, one for each of its two + // lanes, the shared one and the proposals', one after the other + // (flushBudget each); a node with the git role registers its git + // lanes beside them, and render.Bootstrap gives no data pod that + // role; // - 100: the custody keeper's consumer, looked up, created and read // back, then looked up and read back again to attach; // - 10: the fleet broker's registration; // - 35: the state logs' ceilings, read together under LookupBudget, // then the broker's limit at 5 s; // - 120: the state logs' streams (soloSequenceBudget); - // - 500: the join's reads of where the fleet stands: a listing of the - // nodes' positions (25 s: 20 for a listing, and 5 to read back its - // one row), eight of the logs' trim floors (55 s each: 20, and 5 for - // each of the seven floors), and each log's stats at 5 s; + // - 265: the join's reads of where the fleet stands: a listing of the + // nodes' positions (20 s: 15 for a listing, whose key index is one + // page of the bucket's subjects at 5 s, beside its watch's creation + // at 5 s, then a 5 s idle pass and the 5 s stop of the watch's + // consumer; and 5 to read back its one row), every log's trim floor, + // each the one read of its key and all eight at once (5 s, + // readFloors), and each log's stats at 30 s; // - 10: the window for a donor's offer, when the node is behind, and // the 5 s delete of the consumer a listing in it leaves; - // - 130: each state log's consumer (soloSequenceBudget) and the two 5 s - // reads that verify the last one's checkpoint past that window; + // - 180: each state log's consumer (soloSequenceBudget) and the two + // 30 s reads that verify the last one's checkpoint past that window; // - 1: the 25 ms polls of the library lock and the listeners. - // Those counts are the topology's: one data node, seven logs. What - // grows with what the node holds is not counted, as its recovery is - // not: 15 s for each batch of a seat pod's events this node kept and - // could not settle, which the custody keeper claims before it attaches: - // the one in hand at a crash, more only after the coordination store - // stopped answering. What the company configures waits for none of it: - // the engine takes the snapshot of the company's secrets, builds the - // company and restores its coding CLIs' logins after /health answers - // 200. - dataPodBootSeconds = 60 + 30 + 60 + 120 + 60 + 120 + 90 + 40 + 40 + 10 + 100 + 10 + 35 + 120 + 500 + 10 + 130 + 1 + // Those counts are the topology's: one data node, eight logs. A log's + // stats, and the reads that verify its checkpoint, are counted at 30 s: + // the engine asks such a read again while the broker answers none, and a + // boot's context sets no deadline, so the read ends at the ceiling of + // every read asked that way (jsprovision.ReadWith, LookupBudget), where a + // join told nothing fails the boot and an unread checkpoint lets it go + // on. A leader read a step makes on a context with no deadline is asked + // again inside the client's 5 s default instead, as are the listing's + // page and the broker's limit, and each is counted at that. What grows + // with what the node holds is not counted, as its recovery is not: 15 s + // for each batch of a seat pod's events this node kept and could not + // settle, which the custody keeper claims before it attaches: the one in + // hand at a crash, more only after the coordination store stopped + // answering. What the company configures waits for none of it: the + // engine takes the snapshot of the company's secrets, builds the company + // and restores its coding CLIs' logins after /health answers 200. + dataPodBootSeconds = 60 + 30 + 60 + 120 + 60 + 120 + 85 + 40 + 45 + 20 + 100 + 10 + 35 + 120 + 265 + 10 + 180 + 1 // dataPodNATSObjectsSeconds is the object store's step on the default // backend: the broker's limit read (LookupBudget, 30 s), then the bucket // looked up (30 s), created (30 s, soloBudget), read back and bound @@ -740,6 +755,39 @@ type engineEnvironment struct { bridgeURL string } +// receiverURL is CREWLET_SANDBOX_OTEL_RECEIVER_URL for a company's pods, or "" +// for none: Crewlet's sandbox telemetry receiver, to which a remote sandbox +// exports at the engine's public listener, the one port it can reach, with a +// token signed for its run, and which forwards to the class's collector with +// its headers, which never enter the box. Every pod is given it, a seat pod's +// too: the pod that launches a run mints its endpoint, under the company's +// address, and the data pod that address reaches verifies it with the keyring +// every pod shares. Without a collector there is nothing to forward to, and +// without a public address nothing a sandbox could reach. +func receiverURL(c *crewletaiv1alpha1.Company, class *crewletaiv1alpha1.EngineClass) string { + if class.Spec.Telemetry.OTLPEndpoint == "" { + return "" + } + return c.Status.Endpoints.PublicBaseURL +} + +// boxURLs are the addresses a data pod's environment hands its sandboxes, as +// engineEnv sets them: bridgeURL as the tool bridge's, where the engine serves +// one, and the telemetry receiver's (receiverURL). Crewlet refuses to boot on +// a value a sandbox cannot dial, which the render judges with the pod's other +// addresses. A seat pod is handed the receiver's alone, the same value, so the +// data pods' verdict holds its pods back too. +func boxURLs(c *crewletaiv1alpha1.Company, class *crewletaiv1alpha1.EngineClass, bridgeURL string) []render.BoxURL { + var out []render.BoxURL + if bridgeURL != "" { + out = append(out, render.BoxURL{Variable: envMCPBridgeURL, URL: bridgeURL}) + } + if receiver := receiverURL(c, class); receiver != "" { + out = append(out, render.BoxURL{Variable: envOTelReceiverURL, URL: receiver}) + } + return out +} + // engineEnv is an engine container's environment: what the bootstrap // document references, then the operator's own settings, then the class's // and the Company's variables. Kubernetes expands $(VAR) only from variables @@ -783,17 +831,8 @@ func engineEnv( if t.HeadersSecretRef.Name != "" { env = append(env, fromSecret(envOTLPHeaders, e.secret, platformKeyOTLPHeaders)) } - // Crewlet's sandbox telemetry receiver: a remote sandbox exports to - // the engine's public listener, the one port it can reach, with a - // token signed for its run, and the engine forwards to the - // collector above with its headers, which never enter the box. - // Every pod is given it, a seat pod's too: the pod that launches a - // run mints its endpoint, under the company's address, and the data - // pod that address reaches verifies it with the keyring every pod - // shares. Without a collector there is nothing to forward to, and - // without a public address nothing a sandbox could reach. - if public := c.Status.Endpoints.PublicBaseURL; public != "" { - env = append(env, literal(envOTelReceiverURL, public)) + if receiver := receiverURL(c, class); receiver != "" { + env = append(env, literal(envOTelReceiverURL, receiver)) } } diff --git a/internal/controller/company_workload_test.go b/internal/controller/company_workload_test.go index a62c571..a9ff4bb 100644 --- a/internal/controller/company_workload_test.go +++ b/internal/controller/company_workload_test.go @@ -18,7 +18,7 @@ import ( // unless the class names a window, and never past the API's day. func TestDataPodStartupSeconds(t *testing.T) { t.Parallel() - const nats, s3 = 1636, 1776 + const nats, s3 = 1461, 1601 classWith := func(seconds int32, backend crewletaiv1alpha1.ObjectBackend) *crewletaiv1alpha1.EngineClass { class := &crewletaiv1alpha1.EngineClass{} class.Spec.Data.StartupSeconds = seconds @@ -34,16 +34,16 @@ func TestDataPodStartupSeconds(t *testing.T) { }{ {"a volume of no size", unset, "0", nats}, {"a quantity below zero, which no claim has", unset, "-2Gi", nats}, - {"the smallest volume", unset, "12Gi", nats + 12*6}, - {"the smallest volume, objects on the stream", classWith(0, crewletaiv1alpha1.ObjectBackendNATS), "12Gi", nats + 12*6}, - {"the smallest volume, objects in S3", classWith(0, crewletaiv1alpha1.ObjectBackendS3), "9Gi", s3 + 9*6}, + {"the smallest volume", unset, "14Gi", nats + 14*6}, + {"the smallest volume, objects on the stream", classWith(0, crewletaiv1alpha1.ObjectBackendNATS), "14Gi", nats + 14*6}, + {"the smallest volume, objects in S3", classWith(0, crewletaiv1alpha1.ObjectBackendS3), "10Gi", s3 + 10*6}, {"a volume of no size, objects in S3", classWith(0, crewletaiv1alpha1.ObjectBackendS3), "0", s3}, {"a volume of whole GiB", unset, "100Gi", nats + 100*6}, {"a part of a GiB counts as one", unset, "10737418241", nats + 11*6}, {"a volume in decimal units", unset, "500G", nats + 466*6}, {"a volume whose window would pass a day", unset, "20Ti", maxStartupSeconds}, - {"the largest volume whose window is a day", unset, "14127Gi", nats + 14127*6}, - {"a GiB more", unset, "14128Gi", maxStartupSeconds}, + {"the largest volume whose window is within a day", unset, "14156Gi", nats + 14156*6}, + {"a GiB more", unset, "14157Gi", maxStartupSeconds}, {"a volume whose Value saturates", unset, "16Ei", maxStartupSeconds}, {"a volume whose Value wraps below zero", unset, "9223372036854775808", maxStartupSeconds}, {"a volume whose Value reads zero", unset, "1e30", maxStartupSeconds}, @@ -64,7 +64,7 @@ func TestDataPodStartupSeconds(t *testing.T) { // of a period more. func TestStartupFailureThreshold(t *testing.T) { t.Parallel() - for _, seconds := range []int32{1142, 1192, 1636, 1637, 1640, 1641, 1732, 1776, maxStartupSeconds - 1, maxStartupSeconds} { + for _, seconds := range []int32{1137, 1187, 1461, 1462, 1465, 1466, 1557, 1601, maxStartupSeconds - 1, maxStartupSeconds} { threshold := startupFailureThreshold(seconds) if window := (threshold - 1) * probePeriodSeconds; window < seconds || window >= seconds+probePeriodSeconds { t.Errorf("startupFailureThreshold(%d) = %d, a window of %ds after the first probe", seconds, threshold, window) @@ -73,15 +73,15 @@ func TestStartupFailureThreshold(t *testing.T) { } // TestSeatPodStartupSeconds holds a seat pod's window to the engine's bounds -// on a leaf's boot as read from its source, which the mirrors name: 952 +// on a leaf's boot as read from its source, which the mirrors name: 947 // seconds before its object store, and the store's 190 on the event stream // or the 240 of the probe of an S3 bucket. func TestSeatPodStartupSeconds(t *testing.T) { t.Parallel() for backend, want := range map[crewletaiv1alpha1.ObjectBackend]int32{ - "": 1142, - crewletaiv1alpha1.ObjectBackendNATS: 1142, - crewletaiv1alpha1.ObjectBackendS3: 1192, + "": 1137, + crewletaiv1alpha1.ObjectBackendNATS: 1137, + crewletaiv1alpha1.ObjectBackendS3: 1187, } { class := &crewletaiv1alpha1.EngineClass{} class.Spec.Objects.Backend = backend diff --git a/internal/controller/config_optional_test.go b/internal/controller/config_optional_test.go index 85e8482..8f304c0 100644 --- a/internal/controller/config_optional_test.go +++ b/internal/controller/config_optional_test.go @@ -245,7 +245,7 @@ func TestAnEmptyValueOfTheCompanysOwnStepIsSaid(t *testing.T) { valueSecret(stepSecret, map[string]string{stepKey: ""})) step := crewletaiv1alpha1.SandboxSetupStep{ Name: "registry", - Env: []crewletaiv1alpha1.SecretEnvVar{{Name: stepVar, ValueFrom: fromSecret(stepSecret, stepKey)}}, + Env: []crewletaiv1alpha1.SandboxEnvVar{{Name: stepVar, ValueFrom: fromSecret(stepSecret, stepKey)}}, } if c.optional { step.OptionalKeys = []string{stepVar} diff --git a/internal/controller/config_pause.go b/internal/controller/config_pause.go index c2691e1..def2d0e 100644 --- a/internal/controller/config_pause.go +++ b/internal/controller/config_pause.go @@ -227,7 +227,7 @@ func reported(state engine.SeatPause) pauseVerdict { } message += ". It starts no new turn and its mail waits on its inbox" if state.StopRunning { - message += "; the turn it was on was stopped" + message += "; the turn it was on was stopped, and its detached coding runs' writes are refused until it is resumed" } return pauseVerdict{ status: metav1.ConditionTrue, reason: crewletaiv1alpha1.ReasonSeatPaused, message: message + ".", pause: state, @@ -237,7 +237,7 @@ func reported(state engine.SeatPause) pauseVerdict { func pausedWord(paused, stop bool) string { switch { case paused && stop: - return "paused, its running turn stopped" + return "paused, its running turn stopped and its runs' writes refused" case paused: return "paused" } diff --git a/internal/controller/config_pause_test.go b/internal/controller/config_pause_test.go index ac9fd94..849f638 100644 --- a/internal/controller/config_pause_test.go +++ b/internal/controller/config_pause_test.go @@ -5,6 +5,7 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "strings" "sync" "testing" "time" @@ -176,3 +177,31 @@ func TestASeatTheEngineDoesNotRunHasNoPause(t *testing.T) { t.Errorf("the operator called %v for a seat the engine does not run", f.calls) } } + +// TestAStoppedSeatSaysItsRunsWritesAreRefused holds the Paused condition of a +// seat paused with stop_running to everything the stop does: the engine ends +// the turn the seat was on and refuses its detached coding runs' writes until +// it is resumed, so a condition naming only the turn would tell the person +// reading it that a run's writes still land. A pause without the stop names +// neither, since it stops new work only. +func TestAStoppedSeatSaysItsRunsWritesAreRefused(t *testing.T) { + at := time.Date(2026, 10, 31, 9, 0, 0, 0, time.UTC) + for _, tc := range []struct { + name string + stop bool + }{{"stopped", true}, {"paused only", false}} { + t.Run(tc.name, func(t *testing.T) { + message := reported(engine.SeatPause{Paused: true, By: "jane", At: at, StopRunning: tc.stop}).message + for _, phrase := range []string{"the turn it was on was stopped", "detached coding runs' writes are refused until it is resumed"} { + if strings.Contains(message, phrase) != tc.stop { + t.Errorf("the condition's message %q names %q: %v, want %v", message, phrase, !tc.stop, tc.stop) + } + } + word := pausedWord(true, tc.stop) + if strings.Contains(word, "writes refused") != tc.stop { + t.Errorf("pausedWord(true, %v) = %q: says the runs' writes are refused is %v, want %v", + tc.stop, word, !tc.stop, tc.stop) + } + }) + } +} diff --git a/internal/engine/enginetest/enginetest.go b/internal/engine/enginetest/enginetest.go index b426576..356cf9e 100644 --- a/internal/engine/enginetest/enginetest.go +++ b/internal/engine/enginetest/enginetest.go @@ -444,11 +444,11 @@ func (e *Engine) stop(t T, cmd *exec.Cmd, exited <-chan error) { } } -// streamMaxBytes is the embedded broker's declared limit: 9 GiB, the share -// the render gives the smallest data volume an EngineClass that keeps its -// files in the stream's bucket admits (three quarters of 12Gi), as every -// engine here does. A fresh node reserves at least 9305762474 bytes of it -// whatever it runs (Crewlet's engine.BrokerFloor: its seven state logs' +// streamMaxBytes is the embedded broker's declared limit: 10.5 GiB, the +// share the render gives the smallest data volume an EngineClass that keeps +// its files in the stream's bucket admits (three quarters of 14Gi), as every +// engine here does. A fresh node reserves at least 10737418239 bytes of it +// whatever it runs (Crewlet's engine.BrokerFloor: its eight state logs' // ceilings at their smallest and the object store's bucket, a quarter of the // limit) and refuses to boot on less, and what is left holds the mailboxes // and the coordination buckets. A suite runs many engines at @@ -459,7 +459,7 @@ func (e *Engine) stop(t T, cmd *exec.Cmd, exited <-chan error) { // never applies a document. A declared limit is a reservation the broker // keeps count of, not a check against the disk, and a test's streams hold // kilobytes. -const streamMaxBytes = 9 << 30 +const streamMaxBytes = 14 << 30 / 4 * 3 // tierA is the engine's bootstrap: the embedded broker with its own store, the // local store and coordination of a single node, an encrypted secret store, @@ -560,6 +560,54 @@ func Binary(t T) string { return bin } +// BoxURLVerdicts is whether the engine refuses to boot on each of variables, +// the variables of a node's environment that hand its sandboxes an address to +// dial back to it (Crewlet's engine.ReadBoxURLs: the tool bridge's, the +// telemetry receiver's and the code host's, CREWLET_BOX_GIT_URL), as env sets +// them: true where it refuses the value, false where it takes it. No pod the +// operator runs carries the third, since the operator does not offer the code +// host and the API refuses every CREWLET_ variable in a Company's or a +// class's env, so its suites ask about the first two alone. A boot reads +// them before it opens anything, and no validation does, since they are the +// pod's environment rather than its configuration; `crewlet llm doctor` reads +// them as the boot does, and says which it would refuse, without running an +// engine. So that is what this asks, beside document, a company document +// with a providers.llm entry for doctor to examine, with env alone as its +// environment (and PATH and HOME) and no Tier A, so that it resolves every +// reference from env, and without its real completions. Doctor's own verdict +// on the entry is not read: on a machine without the entry's CLI it reports a +// problem with it, as it does on a box URL it refuses, and exits with an +// error for either. +func BoxURLVerdicts(t T, document []byte, env map[string]string, variables ...string) map[string]bool { + t.Helper() + dir := t.TempDir() + company := filepath.Join(dir, "company.yaml") + writeFile(t, company, document) + cmd := exec.Command(Binary(t), "llm", "doctor", "-no-smoke", + "-company", company, "-config", filepath.Join(dir, "absent.yaml")) + cmd.Dir = dir + cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + dir} + for name, value := range env { + cmd.Env = append(cmd.Env, name+"="+value) + } + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + err := cmd.Run() + if exit := (*exec.ExitError)(nil); err != nil && !errors.As(err, &exit) { + t.Fatalf("running crewlet llm doctor: %v", err) + } + if stdout.Len() == 0 { + t.Fatalf("crewlet llm doctor printed no report (%v):\n%s", err, stderr.String()) + } + // Each refusal is a problem of the node's, ahead of every entry's, and + // starts with the variable it names, never with its value. + verdicts := make(map[string]bool, len(variables)) + for _, variable := range variables { + verdicts[variable] = bytes.Contains(stdout.Bytes(), []byte("\n - "+variable+" ")) + } + return verdicts +} + // Module is the Go module the crewlet binary is built from. const Module = "github.com/crewlet/crewlet" diff --git a/internal/engine/seats.go b/internal/engine/seats.go index e325709..92c11a4 100644 --- a/internal/engine/seats.go +++ b/internal/engine/seats.go @@ -62,7 +62,8 @@ type SeatPause struct { // Reason is why, in the pauser's words; empty when none was given. Reason string // StopRunning is whether the pause also ended the turn the seat was on - // rather than letting it finish. + // rather than letting it finish, and refuses its detached coding runs' + // writes until the seat is resumed. StopRunning bool } @@ -137,8 +138,10 @@ func (c *Client) SeatPauses(ctx context.Context) (map[string]SeatPause, error) { // PauseSeat pauses the agent seat handle: it starts no new turn until it is // resumed, its mail waits on its inbox in order, and its scheduled runs are -// skipped. stopRunning also ends the turn it is on at its next round; reason -// is shown on the seat and in its feed, at most MaxPauseReasonRunes +// skipped. stopRunning also ends the turn it is on at its next round, and +// refuses every change its detached coding runs would make (a call of the +// seat's tools that writes, a push from their boxes) until it is resumed; +// reason is shown on the seat and in its feed, at most MaxPauseReasonRunes // characters. The pause is recorded as the operator token's. // // Pausing a paused seat changes nothing, unless stopRunning adds a stop the diff --git a/internal/engine/types.go b/internal/engine/types.go index 1169bcf..a2471dc 100644 --- a/internal/engine/types.go +++ b/internal/engine/types.go @@ -352,8 +352,10 @@ type Fleet struct { // claims a few seats every five seconds. Unplaceable []string - // UnmannedRoles are the node roles (data, ingress, seats, workers) no - // live node runs. + // UnmannedRoles are the node roles the company needs that no live node + // runs: data, ingress, seats and workers always, and git only for a + // company that hosts its repositories in the engine (code.native). No + // pod the operator runs holds the git role. UnmannedRoles []string // TargetEpoch is the activation epoch the fleet converges on, zero when @@ -408,7 +410,9 @@ type FleetNode struct { // the pod's name. ID string - // Roles are the node's roles: data, ingress, seats, workers. + // Roles are the node's roles, of the engine's data, ingress, seats, + // workers and git: a data pod the operator runs holds the first four, + // and a seat pod seats alone. Roles []string // Broker is how the node's broker takes part in the fleet's. diff --git a/internal/render/address.go b/internal/render/address.go index d71b90d..377f32d 100644 --- a/internal/render/address.go +++ b/internal/render/address.go @@ -9,6 +9,7 @@ import ( "strconv" "strings" "unicode" + "unicode/utf8" ) // The Tier A keys of the addresses a data pod's engine is told it is reached @@ -20,6 +21,26 @@ const ( FieldArtifactsURL = "api.artifacts.external_url" ) +// The variables of an engine pod's environment that hand an address of the +// engine to its sandboxes, which dial it back (Crewlet's mcpbridge.BaseURLVar, +// the tool bridge's, and sandbox.OtelReceiverURLVar, the telemetry +// receiver's), and which AddressError names when Crewlet would refuse to +// start on the address one holds. Crewlet's boot reads a third the same way, +// sandbox.BoxGitURLVar (CREWLET_BOX_GIT_URL), the base a coding box reaches +// the company's code host at, which no pod the operator runs carries: the +// operator does not offer the code host, so it sets none, and the API refuses +// every CREWLET_ variable a Company or a class gives the engine's pods. +const ( + EnvBridgeURL = "CREWLET_MCP_BRIDGE_URL" + EnvReceiverURL = "CREWLET_SANDBOX_OTEL_RECEIVER_URL" +) + +// BoxURL is an address an engine pod's environment hands its sandboxes: the +// variable, EnvBridgeURL or EnvReceiverURL, and the address it holds. +type BoxURL struct { + Variable, URL string +} + // ArtifactPlaceholder names an artifact's host label in an artifact origin // (Crewlet's config.ArtifactPlaceholder). const ArtifactPlaceholder = "{artifact}" @@ -35,21 +56,24 @@ const ( const artifactLabelLen = 26 // ErrAddressRefused is a data pod's Tier A asked for with an address Crewlet -// refuses to start on (an *AddressError says which). The API refuses such an -// address where a Company or its class states it, but the operator also -// derives addresses from objects it does not own: the API route's host is the -// class's template expanded with the Company's namespace and name, and a -// namespace named 4711 or 0x1f under {{name}}.{{namespace}}, or a Company -// named cafe under {{uid}}.0x{{name}}, makes a host whose last label is a -// number, which Crewlet refuses as a browser's address. So the render refuses -// such an address first, and the caller holds the pods back on it rather than -// start one that crash-loops. -var ErrAddressRefused = errors.New("crewlet refuses to start on an address the data pod's crewlet.yaml names") +// refuses to start on (an *AddressError says which), in its crewlet.yaml or in +// the variables that hand it to sandboxes (BootstrapInput.BoxURLs). The API +// refuses such an address where a Company or its class states it, but the +// operator also derives addresses from objects it does not own: the API +// route's host, and the webhooks route's, are the class's templates expanded +// with the Company's namespace and name, and a namespace named 4711 or 0x1f +// under {{name}}.{{namespace}}, or a Company named cafe under +// {{uid}}.0x{{name}}, makes a host whose last label is a number, which +// Crewlet refuses as a browser's address and as one a sandbox dials. So the +// render refuses such an address first, and the caller holds the pods back on +// it rather than start one that crash-loops. +var ErrAddressRefused = errors.New("crewlet refuses to start on an address the data pod's crewlet.yaml or environment names") // AddressError is the address Crewlet refuses, and why. type AddressError struct { - // Field is the Tier A key the address is written to: FieldExternalURL, - // FieldPublicURL, FieldArtifactsURL or FieldAllowedOrigins. + // Field is where the address is written: the Tier A key, FieldExternalURL, + // FieldPublicURL, FieldArtifactsURL or FieldAllowedOrigins, or the + // variable that hands it to sandboxes, EnvBridgeURL or EnvReceiverURL. Field string // Address is the address as the render was given it. Address string @@ -78,8 +102,11 @@ func (e *AddressError) Unwrap() error { return ErrAddressRefused } // ${NAME} reference, which the engine would resolve before it judges the // address, is not read as one: the API refuses one in every address a // Company states, and the operator derives none. An artifact origin is judged -// last, since its verdict reads the other addresses. -func refusedAddress(external, public, artifacts string, origins []string) error { +// after them, since its verdict reads the other addresses, and then each +// address handed to sandboxes, as the engine's boot reads it +// (engine.ReadBoxURLs): by sandbox.ParseBoxURL's grammar, which a value that +// is set has to meet whatever the company runs. +func refusedAddress(external, public, artifacts string, origins []string, boxes []BoxURL) error { if reason := externalURLRefusal(strings.TrimSpace(external), true); reason != "" { return &AddressError{Field: FieldExternalURL, Address: external, Reason: reason} } @@ -91,15 +118,189 @@ func refusedAddress(external, public, artifacts string, origins []string) error return &AddressError{Field: FieldAllowedOrigins, Address: origin, Reason: reason} } } - if artifacts == "" { - return nil + if artifacts != "" { + if reason := artifactsURLRefusal(artifacts, external, public, origins); reason != "" { + return &AddressError{Field: FieldArtifactsURL, Address: artifacts, Reason: reason} + } } - if reason := artifactsURLRefusal(artifacts, external, public, origins); reason != "" { - return &AddressError{Field: FieldArtifactsURL, Address: artifacts, Reason: reason} + for _, box := range boxes { + if reason := boxURLRefusal(box.URL); reason != "" { + return &AddressError{Field: box.Variable, Address: box.URL, Reason: reason} + } } return nil } +// boxURLRefusal is why Crewlet refuses to boot with raw as an address a +// sandbox dials back to the engine (sandbox.ParseBoxURL), or "": it must be +// an absolute http or https URL, with no user information, a host a +// sandbox's URL parser takes (boxHostRefusal), a port from 1 to 65535 where +// it names one, and no query or fragment, since the engine appends its own +// path to it. White space around it is dropped, and a value of white space +// alone is unset, a feature the engine turns off rather than refuses, which +// the operator never hands a pod. +func boxURLRefusal(raw string) string { + value := strings.TrimSpace(raw) + if value == "" { + return "" + } + scheme, _, absolute := strings.Cut(value, "://") + if !absolute { + return "is not an absolute URL, with a scheme and a host, which a sandbox dials as it is written" + } + if !strings.EqualFold(scheme, schemeHTTP) && !strings.EqualFold(scheme, schemeHTTPS) { + return "names a scheme a sandbox cannot dial: it must be http or https" + } + u, err := url.Parse(value) + if err != nil { + return "does not parse as a URL" + } + if u.User != nil { + return "carries user information, a name or password before an @, which every sandbox would be handed" + } + if u.Hostname() == "" { + return "names no host for a sandbox to dial" + } + if reason := boxHostRefusal(u); reason != "" { + return reason + } + if port := u.Port(); port != "" { + if n, err := strconv.ParseUint(port, 10, 16); err != nil || n == 0 { + return fmt.Sprintf("has port %s, and a port is 1 to 65535", port) + } + } + if u.RawQuery != "" || u.ForceQuery { + return "carries a query, which the path the engine appends to it would follow" + } + // By the character, because url.Parse records "https://h#" as no + // fragment at all, and a client sends nothing after a '#'. + if strings.Contains(value, "#") { + return "carries a fragment, after which a client sends none of the path the engine appends to it" + } + return "" +} + +// boxHostRefusal is why Crewlet refuses the host url.Parse read out of an +// address a sandbox dials (sandbox's boxHostRefusal), or "": the host the URL +// parser of a sandbox's coding agent takes, the WHATWG URL Standard's, which +// refuses an IPv6 zone, a host that is not UTF-8 once its escapes are +// decoded, one holding a forbidden domain code point, and one whose last +// label is a number and that is no IPv4 address. Net/url, the API server's +// URL parser and the engine's Tier A check, takes each. +func boxHostRefusal(u *url.URL) string { + host := u.Hostname() + if strings.HasPrefix(u.Host, "[") { + if strings.Contains(host, "%") { + return "names an IPv6 host with a zone, which the URL parser of a sandbox's coding agent refuses" + } + return "" + } + if !utf8.ValidString(host) { + return "names a host that is not valid UTF-8, a % escape in it decoding to bytes that spell no character, " + + "which the URL parser of a sandbox's coding agent refuses" + } + if strings.ContainsFunc(host, forbiddenInADomain) { + return "names a host holding a character no host may hold, a space, a control character or one of " + + "# % / : < > ? @ [ \\ ] ^ |, which the URL parser of a sandbox's coding agent refuses" + } + if endsInANumber(host) && !isIPv4Address(host) { + return "names a host whose last label is a number, which the URL parser of a sandbox's coding agent reads " + + "as an IPv4 address, and it is none: write an address as four numbers from 0 to 255 joined by dots, " + + "and a host name whose last label is not a number" + } + return "" +} + +// forbiddenInADomain is the WHATWG URL Standard's forbidden domain code point +// (sandbox's forbiddenInADomain): a C0 control, space, DEL, or one of +// # % / : < > ? @ [ \ ] ^ |. +func forbiddenInADomain(r rune) bool { + return r <= ' ' || r == 0x7f || strings.ContainsRune(`#%/:<>?@[\]^|`, r) +} + +// endsInANumber is the standard's ends-in-a-number checker (sandbox's +// endsInANumber): whether a name's last label, one trailing dot ignored, is +// decimal digits, or any other part ipv4Number reads. +func endsInANumber(name string) bool { + labels := strings.Split(name, ".") + if labels[len(labels)-1] == "" { + if len(labels) == 1 { + return false + } + labels = labels[:len(labels)-1] + } + last := labels[len(labels)-1] + if last != "" && strings.Trim(last, "0123456789") == "" { + return true + } + _, ok := ipv4Number(last) + return ok +} + +// isIPv4Address is whether the standard's IPv4 parser reads name as an +// address (sandbox's isIPv4Address): at most four dot-separated parts, one +// trailing dot ignored, each a number ipv4Number reads, every one but the last +// at most 255 and the last below 256 to the power of the bytes left for it. +func isIPv4Address(name string) bool { + parts := strings.Split(name, ".") + if len(parts) > 1 && parts[len(parts)-1] == "" { + parts = parts[:len(parts)-1] + } + if len(parts) > 4 { + return false + } + for i, part := range parts { + n, ok := ipv4Number(part) + if !ok { + return false + } + if i < len(parts)-1 && n > 255 { + return false + } + if i == len(parts)-1 && n >= 1<<(8*(5-len(parts))) { + return false + } + } + return true +} + +// ipv4Number is the standard's IPv4 number parser (sandbox's ipv4Number): a +// part in hexadecimal after 0x or 0X, in octal after a leading 0, and in +// decimal otherwise, 0x alone being zero; false for an empty part or a digit +// outside its radix. The value stops at 2^32, past every bound an address +// puts on a part. +func ipv4Number(part string) (n uint64, ok bool) { + if part == "" { + return 0, false + } + radix := uint64(10) + switch { + case len(part) >= 2 && (part[:2] == "0x" || part[:2] == "0X"): + part, radix = part[2:], 16 + case len(part) >= 2 && part[0] == '0': + part, radix = part[1:], 8 + } + const saturated = 1 << 32 + for i := range len(part) { + var d uint64 + switch c := part[i]; { + case '0' <= c && c <= '9': + d = uint64(c - '0') + case 'a' <= c && c <= 'f': + d = uint64(c-'a') + 10 + case 'A' <= c && c <= 'F': + d = uint64(c-'A') + 10 + default: + return 0, false + } + if d >= radix { + return 0, false + } + n = min(n*radix+d, saturated) + } + return n, true +} + // artifactsURLRefusal is why Crewlet refuses raw as api.artifacts.external_url // beside the node's other addresses (config.APIArtifacts.validate), or "". func artifactsURLRefusal(raw, external, public string, origins []string) string { diff --git a/internal/render/bootstrap.go b/internal/render/bootstrap.go index ad99d8f..0c70f24 100644 --- a/internal/render/bootstrap.go +++ b/internal/render/bootstrap.go @@ -73,9 +73,10 @@ const ( // LeafPort is the port of a data pod's leaf listener, which the seat // pods join the company's event stream through. It is open only when // the class's seatPools.policy is Allowed. Like every port of - // Crewlet's embedded broker it accepts any connection that reaches it, - // with no authentication and no TLS, so only a NetworkPolicy keeps - // another tenant's pods off it. + // Crewlet's embedded broker it admits a link only on a key derived from + // the company's keyring, which every pod's crewlet.yaml carries, and + // encrypts nothing; a NetworkPolicy keeps every other pod from + // reaching it at all. LeafPort = 7422 ) @@ -139,6 +140,11 @@ type BootstrapInput struct { // ArtifactsURL is Crewlet's api.artifacts.external_url, the Company's // status.endpoints.artifactsBaseURL. Empty opens no artifact listener. ArtifactsURL string + // BoxURLs are the addresses the pod's environment hands its sandboxes, + // which no crewlet.yaml names: each variable the caller sets, with its + // value. Crewlet refuses to boot on one a sandbox cannot dial, so they + // are judged with the addresses above, and the pods held back on one. + BoxURLs []BoxURL } // ErrNoAddress is a data pod's Tier A asked for without an address the engine @@ -431,7 +437,7 @@ func Bootstrap(in *BootstrapInput) (*BootstrapDocument, error) { if in.ExternalURL == "" || in.PublicURL == "" { return nil, fmt.Errorf("%w: api.external_url is %q and api.public.external_url %q", ErrNoAddress, in.ExternalURL, in.PublicURL) } - if err := refusedAddress(in.ExternalURL, in.PublicURL, in.ArtifactsURL, in.AllowedOrigins); err != nil { + if err := refusedAddress(in.ExternalURL, in.PublicURL, in.ArtifactsURL, in.AllowedOrigins, in.BoxURLs); err != nil { return nil, err } signIn := in.Class.Spec.SignIn diff --git a/internal/render/bootstrap_test.go b/internal/render/bootstrap_test.go index e12c24c..371ae32 100644 --- a/internal/render/bootstrap_test.go +++ b/internal/render/bootstrap_test.go @@ -469,6 +469,67 @@ func TestBootstrapNeedsTheEnginesAddresses(t *testing.T) { } } +// boxURLTaken and boxURLRefused are addresses handed to sandboxes either side +// of each line Crewlet's grammar for them draws (sandbox.ParseBoxURL), from +// the engine's own cases of it: an absolute http or https URL with a host a +// sandbox's URL parser takes, the WHATWG URL Standard's, which reads the IPv4 +// short forms and refuses an IPv6 zone, a forbidden domain code point, a host +// that is not UTF-8 and a name ending in a number that is no IPv4 address; +// a port from 1 to 65535; and no user information, query or fragment. White +// space alone is a value the engine reads as unset. TestContractBoxURLShapes +// asks the engine about these and more. +var ( + boxURLTaken = []string{ + "", " \t", "https://engine.example.com/", "http://172.17.0.1:8080/base/", "http://engine.example.com", + " https://engine.example.com:8443 ", "HTTPS://engine.example.com", "http://[fd00::1]:8000", + "https://ex%C3%A9.example/", "https://ex\u00e9.example", "http://engine.example.com/a%2Fb/", + "http://engine.example.com:1", "http://engine.example.com:65535", "http://engine.example.com:/base", + "http://127.1:8000", "http://0x7f.1", "http://10.0.0.1.:8000", "http://4294967295", "http://engine.1a", + "http://engine.0xg", + } + boxURLRefused = []string{ + "engine.example.com:80", "127.0.0.1:8080", "engine.example.com", "http:engine.example.com", + "ftp://engine.example.com", "http://", "http:///x", "http://:8080", "http://[fe80::1%25eth0]:8000", + "http://engine%25a.example.com", "http://:8000", "http://engine]a.example.com", + "https://%FF.example.com", "http://10.0.0.300:8000", "http://256.0.0.1:8000", "http://10.0.0.1.2", + "http://4294967296", "http://10..0.1", "http://08.0.0.1", "http://engine.example.123", + "http://engine.example.123.", "http://engine.0X1F", "http://engine.example.com:0", + "http://engine.example.com:99999", "http://[fd00::1]:65536", "https://ops:hunter2@engine.example.com", + "https://@engine.example.com", "https://engine.example.com/?region=eu", "https://engine.example.com/?", + "https://engine.example.com/#otlp", "https://engine.example.com#", "https://engine example.com", + "https://ops:%zz@engine.example.com", + } +) + +// TestBootstrapRefusesABoxURLNoSandboxCanDial: Crewlet refuses to boot on an +// address handed to sandboxes that their URL parser cannot read, which no +// crewlet.yaml names, so the render refuses one the caller hands it in either +// variable with an AddressError naming the variable and the address, and +// takes the rest, which its own addresses leave as they are. +func TestBootstrapRefusesABoxURLNoSandboxCanDial(t *testing.T) { + for _, variable := range []string{EnvBridgeURL, EnvReceiverURL} { + for _, c := range []struct { + shapes []string + refused bool + }{{boxURLTaken, false}, {boxURLRefused, true}} { + for _, shape := range c.shapes { + t.Run(fmt.Sprintf("%s/%q", variable, shape), func(t *testing.T) { + in := bootstrapCases()[0].input(t) + in.BoxURLs = []BoxURL{{Variable: variable, URL: shape}} + _, err := Bootstrap(in) + refused, ok := errors.AsType[*AddressError](err) + switch { + case !c.refused && err != nil: + t.Fatalf("Bootstrap() refused %s %q: %v", variable, shape, err) + case c.refused && (!ok || refused.Field != variable || refused.Address != shape): + t.Fatalf("Bootstrap() = %v, want an AddressError for %s %q", err, variable, shape) + } + }) + } + } + } +} + // loopbackCases are addresses a data pod's engine may be told people reach it // at, each with whether Crewlet reads it as on loopback, and so lets a second // factor be optional there without the acknowledgement. TestLoopback holds diff --git a/internal/render/codework_test.go b/internal/render/codework_test.go index 49ff07b..1335033 100644 --- a/internal/render/codework_test.go +++ b/internal/render/codework_test.go @@ -153,7 +153,7 @@ func TestCodeWork(t *testing.T) { } setupEnv := func(company bool, name string) func(*Input) { return func(in *Input) { - step := v1alpha1.SandboxSetupStep{Name: "login", Env: []v1alpha1.SecretEnvVar{{ + step := v1alpha1.SandboxSetupStep{Name: "login", Env: []v1alpha1.SandboxEnvVar{{ Name: name, ValueFrom: v1alpha1.SecretValueSource{SecretKeyRef: v1alpha1.SecretKeyReference{Name: "setup", Key: keyValue}}, }}} diff --git a/internal/render/company.go b/internal/render/company.go index 0ee8b0d..b514bfe 100644 --- a/internal/render/company.go +++ b/internal/render/company.go @@ -486,11 +486,9 @@ func (r *renderer) company() companyDoc { RootSpace: spec.Knowledge.RootSpace, Vectors: spec.Knowledge.Vectors, }, + Inbox: inbox(spec.Inbox), TokenBudget: tokenBudget(spec.TokenBudget), } - if days := spec.Tracker.Worklet.InboxRetentionDays; days != 0 { - doc.Tracker.Worklet = &trackerWorkletDoc{InboxRetentionDays: days} - } doc.Providers = r.providersDoc() doc.Integrations = r.integrations() doc.MCPServers = r.mcpServers() @@ -498,6 +496,16 @@ func (r *renderer) company() companyDoc { return doc } +// secondsPerDay is a day in seconds. +const secondsPerDay = 24 * 60 * 60 + +// inbox is Crewlet's inbox block. The API states the horizon in seconds, as +// it states every duration, and Crewlet keeps it in whole days: the API +// refuses a horizon that is not whole days, so the division is exact. +func inbox(s v1alpha1.InboxSettings) inboxDoc { + return inboxDoc{RetentionDays: s.RetentionSeconds / secondsPerDay} +} + func tokenBudget(b v1alpha1.TokenBudget) tokenBudgetDoc { return tokenBudgetDoc{Day: b.Day, Week: b.Week, Month: b.Month} } diff --git a/internal/render/company_test.go b/internal/render/company_test.go index 473385c..f1679ac 100644 --- a/internal/render/company_test.go +++ b/internal/render/company_test.go @@ -4,10 +4,13 @@ import ( "bytes" "fmt" "path/filepath" + "reflect" "regexp" "slices" + "strings" "testing" + "k8s.io/apimachinery/pkg/util/validation/field" "sigs.k8s.io/yaml" "github.com/crewlet/crewlet-operator/internal/secretref" @@ -25,10 +28,19 @@ type companyCase struct { exposed bool // advisories are the advisories Crewlet's validator gives the // document, about choices of the tenant's that no render avoids: a - // chart of agents alone, which has no human seat. + // chart of agents alone, which has no human seat, and a code access + // tier nothing applies. advisories []advisory } +// advisoryCodeAccessGrantsNothing is Crewlet's advisory on a seat whose +// role.code.access nothing applies (config.roleCodeWarnings): the company +// hosts no code of its own, which the operator never asks of it, and either +// has no GitHub integration or the seat has no GitHub identity of its own, +// the only places a tier caps what the seat may do. The render writes the +// tier the Seat states all the same, as Crewlet runs it. +var advisoryCodeAccessGrantsNothing = advisory{path: "roles[0].code.access", says: "grants nothing"} + func companyCases() []companyCase { agentsAlone := []advisory{advisoryNoHumanSeat} return []companyCase{ @@ -46,6 +58,7 @@ func companyCases() []companyCase { {name: "secret-positions", exposed: true, advisories: agentsAlone}, {name: "cli-agent", exposed: true, advisories: agentsAlone}, {name: "opencode-agent", exposed: true, advisories: agentsAlone}, + {name: "code-access", advisories: []advisory{advisoryNoHumanSeat, advisoryCodeAccessGrantsNothing}}, {name: caseAtlassian, exposed: true}, {name: caseSeatPools}, {name: caseOptionalKeys}, @@ -358,3 +371,91 @@ func TestCompanyLeavesItsInputAlone(t *testing.T) { } } } + +// TestEngineConfigCannotSetAKeyAFieldRenders holds the three lists of the +// top-level keys the API's fields and objects render to one another, each +// both ways: the keys of the company document (companyDoc), the keys Diff +// compares as the API's own (typedKeys), and the keys the CRD's rule on a +// Company's engineConfig refuses, read off the rule (engineConfigReserved). +// The render merges every engineConfig entry into the document as a +// top-level key, trusting the API to refuse each key a field renders: a key +// the rule let through would be replaced by an entry of its name, and a key +// the rule refused that no field renders would be one nothing could set. A +// key typedKeys left out would be compared as a tuning block, so an edit made +// to it through the engine would go unfound; one it listed that no field +// renders would be compared exactly, so the defaults the engine serves for +// it would read as drift. +func TestEngineConfigCannotSetAKeyAFieldRenders(t *testing.T) { + doc := reflect.TypeFor[companyDoc]() + rendered := make([]string, 0, doc.NumField()) + for f := range doc.Fields() { + key, _, _ := strings.Cut(f.Tag.Get("json"), ",") + rendered = append(rendered, key) + } + for _, key := range lacking(typedKeys, rendered) { + t.Errorf("a field of the API renders %s, and typedKeys does not list it: Diff would compare it as a "+ + "tuning block, so an edit made to it through the engine would go unfound", key) + } + for _, key := range lacking(rendered, typedKeys) { + t.Errorf("typedKeys lists %s, which no field of the API renders: Diff would compare it exactly, so the "+ + "defaults the engine serves for it would read as drift", key) + } + reserved := engineConfigReserved(t) + for _, key := range lacking(reserved, rendered) { + t.Errorf("a field of the API renders %s, and the CRD's rule lets engineConfig set it: an entry of that "+ + "name would replace what the field renders; add it to the rule on Company.spec.engineConfig", key) + } + for _, key := range lacking(rendered, reserved) { + t.Errorf("the CRD's rule refuses %s in engineConfig, and no field of the API renders it: nothing could "+ + "set it; remove it from the rule on Company.spec.engineConfig", key) + } + // The keys read off the rule are the ones the API server refuses, with + // the rule's message. + validate := engineConfigValidator(t) + for _, key := range reserved { + errs := validate(map[string]any{keyName: key, "value": map[string]any{}}) + if !slices.ContainsFunc(errs, func(e *field.Error) bool { return strings.HasPrefix(e.Detail, engineConfigRefusal) }) { + t.Errorf("the CRD's rule names %s, and its validator lets engineConfig set it (%v)", key, errs) + } + } +} + +// reservedKeysRule is the rule on Company.spec.engineConfig that refuses the +// keys a field renders, as controller-gen writes it into the CRD, its list of +// quoted keys the first group. The rule that refuses code, a key no field +// renders, has another shape and is not read. +var reservedKeysRule = regexp.MustCompile( + `^self\.all\(e, !\(e\.name in \[('[a-z][a-z0-9_]*'(?:, '[a-z][a-z0-9_]*')*)\]\)\)$`) + +// engineConfigReserved is the keys the CRD's rule on Company.spec.engineConfig +// refuses as ones a field renders, read off the rule. It must be there +// exactly once, in the shape reservedKeysRule reads, or the test fails +// rather than read nothing. +func engineConfigReserved(t *testing.T) []string { + t.Helper() + var lists []string + for _, r := range engineConfigSchema(t).XValidations { + if m := reservedKeysRule.FindStringSubmatch(r.Rule); m != nil { + lists = append(lists, m[1]) + } + } + if len(lists) != 1 { + t.Fatalf("want exactly one rule of the shape %s on Company.spec.engineConfig, found %q", reservedKeysRule, lists) + } + keys := strings.Split(lists[0], ", ") + for i, quoted := range keys { + keys[i] = strings.Trim(quoted, "'") + } + return keys +} + +// lacking is every key of want that keys does not hold. +func lacking(keys, want []string) []string { + var out []string + for _, key := range want { + if !slices.Contains(keys, key) { + out = append(out, key) + } + } + return out +} diff --git a/internal/render/contract_test.go b/internal/render/contract_test.go index 0dda521..2cb71c5 100644 --- a/internal/render/contract_test.go +++ b/internal/render/contract_test.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" "io" + "maps" "net/http" "net/http/httptest" "os" @@ -449,6 +450,56 @@ func TestContractAddressShapes(t *testing.T) { }) } +// TestContractBoxURLShapes holds the render's verdict on an address handed to +// sandboxes (BootstrapInput.BoxURLs) to Crewlet's, both ways, at every shape +// of boxURLTaken, boxURLRefused and addressShapes, in each variable: Bootstrap +// refuses a shape there with an AddressError naming the variable exactly +// where the engine refuses to boot on it, which `crewlet llm doctor` reports +// as its boot would read it (enginetest.BoxURLVerdicts), and renders it +// wherever the engine takes it; so the render holds back no data pod the +// engine would run, and starts none it would refuse to boot. +func TestContractBoxURLShapes(t *testing.T) { + doc, err := Company(companyCaseNamed(t, "minimal-agent").input(t)) + if err != nil { + t.Fatal(err) + } + env := map[string]string{} + for _, k := range doc.Refs { + env[secretsync.StoreName(k)] = dummyCredential(k) + } + shapes := slices.Concat(boxURLTaken, boxURLRefused, addressShapes) + refusedSomewhere := make([]bool, len(shapes)) + t.Run("shapes", func(t *testing.T) { + for i, shape := range shapes { + t.Run(fmt.Sprintf("%.40q", shape), func(t *testing.T) { + t.Parallel() + withShape := maps.Clone(env) + withShape[EnvBridgeURL], withShape[EnvReceiverURL] = shape, shape + engine := enginetest.BoxURLVerdicts(t, doc.YAML, withShape, EnvBridgeURL, EnvReceiverURL) + refusedSomewhere[i] = engine[EnvBridgeURL] || engine[EnvReceiverURL] + for _, variable := range []string{EnvBridgeURL, EnvReceiverURL} { + in := bootstrapCases()[0].input(t) + in.BoxURLs = []BoxURL{{Variable: variable, URL: shape}} + _, err := Bootstrap(in) + refused, ok := errors.AsType[*AddressError](err) + switch { + case err != nil && (!ok || refused.Field != variable || refused.Address != shape): + t.Fatalf("Bootstrap refused %s %q with %v", variable, shape, err) + case (err != nil) != engine[variable]: + t.Errorf("the render refuses %s %q: %v; Crewlet refuses to boot on it: %v", + variable, shape, err != nil, engine[variable]) + } + } + }) + } + }) + // A doctor whose report this test misread would read every shape as one + // the engine takes, and the render's refusals as the only ones. + if !slices.Contains(refusedSomewhere, true) { + t.Error("crewlet llm doctor refused no box URL: its report did not name the variables as this test reads it") + } +} + // artifactShapes are artifact origins either side of each line Crewlet draws, // read beside a dashboard at nimbusAPI and a public listener at nimbusHooks. var artifactShapes = []string{ diff --git a/internal/render/credentials_test.go b/internal/render/credentials_test.go index da0e62e..78798f6 100644 --- a/internal/render/credentials_test.go +++ b/internal/render/credentials_test.go @@ -288,9 +288,9 @@ func TestCredentialPositionsAreTheEnginesMarks(t *testing.T) { // credentials. const engineConfigRefusal = "engineConfig may not set" -// engineConfigValidator runs the CEL rules of Company.spec.engineConfig as -// the generated CRD states them, with the validator the API server runs. -func engineConfigValidator(t *testing.T) func(entries ...any) field.ErrorList { +// engineConfigSchema is Company.spec.engineConfig as the generated CRD +// states it. +func engineConfigSchema(t *testing.T) *apiextensionsv1.JSONSchemaProps { t.Helper() data, err := os.ReadFile(filepath.Join("..", "..", "config", "crd", "bases", "crewlet.ai_companies.yaml")) if err != nil { @@ -313,6 +313,14 @@ func engineConfigValidator(t *testing.T) func(entries ...any) field.ErrorList { if node == nil { t.Fatal("the Company CRD has no v1alpha1 spec.engineConfig") } + return node +} + +// engineConfigValidator runs the CEL rules of Company.spec.engineConfig as +// the generated CRD states them, with the validator the API server runs. +func engineConfigValidator(t *testing.T) func(entries ...any) field.ErrorList { + t.Helper() + node := engineConfigSchema(t) var internal apiextensions.JSONSchemaProps if err := apiextensionsv1.Convert_v1_JSONSchemaProps_To_apiextensions_JSONSchemaProps(node, &internal, nil); err != nil { t.Fatal(err) diff --git a/internal/render/diff.go b/internal/render/diff.go index 374d40e..635e4ae 100644 --- a/internal/render/diff.go +++ b/internal/render/diff.go @@ -12,9 +12,11 @@ import ( // typedKeys are the top-level keys of a company document that fields and // objects of the API render, the same list the API refuses in a Company's -// engineConfig. Every other top-level key is an engineConfig entry. +// engineConfig (TestEngineConfigCannotSetAKeyAFieldRenders holds the two to +// each other and to companyDoc, each both ways). Every other top-level key is +// an engineConfig entry. var typedKeys = []string{ - "name", "mission", "vision", "policies", "timezone", "tracker", "knowledge", "token_budget", + "name", "mission", "vision", "policies", "timezone", "tracker", "knowledge", "inbox", "token_budget", keyProviders, keyIntegrations, keyMCPServers, keyRoles, keyUnits, } diff --git a/internal/render/document.go b/internal/render/document.go index a0714ad..cc0da46 100644 --- a/internal/render/document.go +++ b/internal/render/document.go @@ -14,6 +14,7 @@ type companyDoc struct { Timezone string `json:"timezone,omitempty"` Tracker trackerDoc `json:"tracker,omitzero"` Knowledge knowledgeDoc `json:"knowledge,omitzero"` + Inbox inboxDoc `json:"inbox,omitzero"` TokenBudget tokenBudgetDoc `json:"token_budget,omitzero"` Providers providersDoc `json:"providers,omitzero"` Integrations integrationsDoc `json:"integrations,omitzero"` @@ -23,12 +24,7 @@ type companyDoc struct { } type trackerDoc struct { - Backend string `json:"backend,omitempty"` - Worklet *trackerWorkletDoc `json:"worklet,omitempty"` -} - -type trackerWorkletDoc struct { - InboxRetentionDays int32 `json:"inbox_retention_days,omitempty"` + Backend string `json:"backend,omitempty"` } type knowledgeDoc struct { @@ -39,6 +35,10 @@ type knowledgeDoc struct { Vectors *bool `json:"vectors,omitempty"` } +type inboxDoc struct { + RetentionDays int32 `json:"retention_days,omitempty"` +} + type tokenBudgetDoc struct { Day int64 `json:"day,omitempty"` Week int64 `json:"week,omitempty"` @@ -285,6 +285,7 @@ type roleDoc struct { MCPEnv map[string]map[string]string `json:"mcp_env,omitempty"` Sandbox *roleSandboxDoc `json:"sandbox,omitempty"` Placement *placementDoc `json:"placement,omitempty"` + Code *roleCodeDoc `json:"code,omitempty"` Integrations roleIntegrationsDoc `json:"integrations,omitzero"` Project string `json:"project,omitempty"` Space string `json:"space,omitempty"` @@ -336,6 +337,13 @@ type placementDoc struct { Labels map[string]string `json:"labels"` } +// roleCodeDoc is a role's code access: one tier for every code host the +// engine drives, which a seat's GitHub App is created with and mints its +// tokens at. +type roleCodeDoc struct { + Access string `json:"access"` +} + type roleIntegrationsDoc struct { Slack *roleSlackDoc `json:"slack,omitempty"` GitHub *roleGitHubDoc `json:"github,omitempty"` @@ -354,7 +362,6 @@ type roleSlackDoc struct { } type roleGitHubDoc struct { - Tier string `json:"tier,omitempty"` Repos []string `json:"repos,omitempty"` AppID int64 `json:"app_id,omitempty"` AppSlug string `json:"app_slug,omitempty"` diff --git a/internal/render/enums.go b/internal/render/enums.go index 3ec5bab..635c0f6 100644 --- a/internal/render/enums.go +++ b/internal/render/enums.go @@ -128,13 +128,13 @@ func containerWebhookMode(m v1alpha1.ContainerWebhookMode) string { return "" } -func gitHubAccessTier(t v1alpha1.GitHubAccessTier) string { +func codeAccessTier(t v1alpha1.CodeAccessTier) string { switch t { - case v1alpha1.GitHubAccessReadOnly: + case v1alpha1.CodeAccessReadOnly: return "read_only" - case v1alpha1.GitHubAccessReview: + case v1alpha1.CodeAccessReview: return "review" - case v1alpha1.GitHubAccessFullAccess: + case v1alpha1.CodeAccessFullAccess: return "full_access" } return "" diff --git a/internal/render/mirror_test.go b/internal/render/mirror_test.go index b414232..63d7ef9 100644 --- a/internal/render/mirror_test.go +++ b/internal/render/mirror_test.go @@ -26,9 +26,10 @@ import ( // tool-server API's refusal of a literal value copies and the engine never // applies to a tool server's values, taking any literal there; and whether // the address a data pod's engine is told is on loopback, and whether it is -// one the engine refuses to start on at all, which the render holds a data -// pod back on (and the API refuses where a Company or its class states it) -// and the engine shows only for the addresses a test tries. A contract test +// one the engine refuses to start on at all, or to boot on where a pod hands +// it to sandboxes, which the render holds a data pod back on (and the API +// refuses where a Company or its class states it) and the engine shows only +// for the addresses a test tries. A contract test // cannot hold those to the engine, so the operator // holds the code it read: the digest of each declaration as it was when it // was read against the operator files that copy it. @@ -104,15 +105,73 @@ const ( // in. Launch and LaunchExecutor hand the box the run's model in place of // an API entry's endpoint, which moved no verdict. cliOnlyRead = "v0.1.1-0.20261023214803-bbedf7ecebd7" - // standingInstructionsRead is the engine that hands an agent-mode run the - // seat's system prompt as its standing instructions, on the coding - // agent's append channel, apart from the brief, and names the run by the - // turn's task (LaunchExecutor's System and Label), and frames a wider - // task after a run_sandbox brief alone (Launch's Wider), an agent-mode - // brief being the turn's ask already. None is a verdict: the refusals - // before a box exists and the sign-in that must reach it are as at - // cliOnlyRead. - standingInstructionsRead = "v0.1.1-0.20261025043416-a2634a307854" + // bridgeStopRead is crewlet/crewlet#298's engine whose agent-mode launch + // opens its run's session on the tool bridge with the seat's stop + // (mcpbridge.Session.Stopped), read at the head of its branch, where + // LaunchExecutor was read again. The launch builds the session in + // bridgeSession now, with the fields it set before and the stop check, + // stoppedBy, the one reading the code host refuses a run's box's pushes + // by: a person's pause that stops the seat's running work refuses every + // bridged call of the run to a tool not proven read-only, until the seat + // is resumed, and answers it paused, naming who stopped the seat; a read + // still runs. That is no verdict on a launch: bridgeSession always sets + // the check, so the bridge's refusal of a session without one is never + // met, and the launch refuses what it refused before, in the same order. + // What a stop refuses a run is said where the operator says what a stop + // does, the Seat API's stopRunning and the applying page's seat pauses. + // Both launches were read before at crewlet/crewlet#298's engine whose + // launches hand the run row they open the code holds of their turn + // (v0.1.1-0.20261029165728-4ee3f06a5fde), the run_sandbox launch read + // again since at runReposRead. A turn holds each scratch repository of + // the company's code host it names, so that the host's + // expiry of an idle one leaves it be while the turn may still use it, and + // a run_sandbox launch (Launch) and an agent-mode one (LaunchExecutor) + // alike seal that set from its read until the row they open exists, so + // that a hold another call of the turn takes meanwhile waits, then have + // the row keep it, every later hold reaching the row before the request + // it is taken for (launchHolds). The one refusal the holds add to a + // launch is the row's, of a set past the most one run may carry + // (sandbox.MaxCodeHolds), which only the repositories a turn names reach + // and no document foresees; and a company the operator runs hosts no + // code (its engineConfig may not set code), so its turns hold nothing. + // An agent-mode launch also resolves its run before it opens the run's + // session on the tool bridge, so that the session carries the run's + // model for every bridged call to name (mcpbridge.Session.Model); it + // refuses a node with no bridge before it assembles the run's + // environment, in the words the bridge's empty endpoint was refused in, + // and opens no session for a run executorRun refuses. Before that, both + // launches were read at the engine that hands an agent-mode run the seat's + // system prompt as its standing instructions, on the coding agent's + // append channel, apart from the brief, names the run by the turn's task + // (LaunchExecutor's System and Label), and frames a wider task after a + // run_sandbox brief alone (Launch's Wider), an agent-mode brief being the + // turn's ask already. None of it is a verdict: the head a launch runs + // on, its coding agent, every other refusal before a box exists and the + // sign-in that must reach the box are as at cliOnlyRead. + bridgeStopRead = "v0.1.1-0.20261030230815-ab6d97e67b58" + // runReposRead is crewlet/crewlet#298's engine whose run_sandbox launch + // gives a run the repositories its call names on the company's own code + // host, read at the head of its branch, where the launch was read again. + // The launch takes the call's request where it took its brief, resolves + // the repositories the call names last of its checks, after every + // refusal a document foresees and before the turn's holds are sealed for + // the row, and hands a run given some its credential on the code host, + // the base its clones and pushes reach and git's configuration, in its + // launch's own environment and never on the row (ADR-0063). A company + // the operator runs hosts no code (its engineConfig may not set code), so + // no call of its seats names a repository: the tool offers repos only + // where the node serves the code host, refusing the argument elsewhere + // before the launcher is asked, and the launch refuses a repository + // where the company hosts no code. So none of it is a verdict: the head + // a launch runs on, its coding agent, every other refusal before a box + // exists and the sign-in that must reach the box are as they were where + // both launches were read for the code holds (bridgeStopRead says how). + // The variables that environment carries are the engine's own + // (config.ReservedSandboxEnv), which Crewlet refuses in every env a run's + // environment is assembled from, and so do the API's rules on a Seat's + // spec.sandbox.env and a setup step's env, which test/crd holds to the + // engine (TestReservedRunVariablesFollowTheEngine). + runReposRead = "v0.1.1-0.20261030022059-2887e830ccb5" // refusalByModeRead is the engine whose remote-run refusal is worded by // the entry's auth mode (credentialRefusal): an api-key entry whose key // resolved to nothing is told to give the seat its own key, and an @@ -170,6 +229,23 @@ const ( // first read at. addressShapeRead = "v0.1.1-0.20261019003935-7cf6955f22ea" addressGo = "internal/render/address.go" + // boxURLGo is where the engine judges an address a sandbox dials back to + // it, and engineBoxURLGo where its boot reads every such address and + // refuses one that is set and unusable, both first read at codeHostRead. + boxURLGo = "internal/sandbox/boxurl.go" + engineBoxURLGo = "internal/engine/boxurl.go" + // boxGitURLRead is crewlet/crewlet#298's engine whose boot reads a third + // address a box dials back to the engine, read at the head of its branch: + // the code host's base a coding box clones and pushes through + // (CREWLET_BOX_GIT_URL, sandbox.BoxGitURLVar), through ParseBoxURL as the + // tool bridge's and the telemetry receiver's are, its boot refused on a + // value that is set and unusable. The operator hands its pods the + // bridge's and the receiver's alone, which the render judges: it does not + // offer the code host, so it sets no code host's base, and no pod of its + // carries one another way, since the API refuses every CREWLET_ variable + // a Company or a class gives the engine's pods. So no boot of its meets + // the third address, and no verdict moves. + boxGitURLRead = "v0.1.1-0.20261030022059-2887e830ccb5" // engineClassGo is the EngineClass API, which several decisions copied // from the engine are written into. engineClassGo = "api/v1alpha1/engineclass_types.go" @@ -203,15 +279,170 @@ const ( // engine's own tracker and pages; it moves no window either. bootRecoveryRead = "v0.1.1-0.20261025183322-23ffa9027d77" + // codeHostRead is crewlet/crewlet#298's engine, the code host's, read at + // the head of its branch, where every bound the windows were read from + // that it moved was read again. The code host's log is an eighth state + // log, so a data pod's admission reads eight maintenance windows (admit + // reads one per registered domain, and the register grew). The join + // reads every log's trim floor once, each by its key and all at once + // (readFloors, readFloorsOf, (*FleetStore).Floor), where it listed the + // fleet's floors once for the join and once more for each log + // ((*FleetStore).Floors, gone, and trimFloor, which only a write's fence + // reads now, so no boot waits on it). The object store's fleet record is + // agreed in casRetry's attempts, bounded by their number, not by time, + // where retryRaces (gone) asked again within the client's 5 s default. A + // log's stats and its record at a sequence are read through ReadWith on + // their caller's context, asked again while the broker answers none, + // where each was one request at the client's default. What else moved + // in the declarations read here moved no bound: the queue's + // registrations name how many answers they run at once, the + // provisioning reads hand their value back rather than set it from a + // closure, the nats bucket's uploads share a window sized for the link + // they cross, the S3 client counts into the node's metrics, and a log's + // start words one refusal anew. A bound moved outside them: the change + // that gave the registrations their answer counts also gave the estate + // service a second lane, for a push's proposals, which a data node + // registers beside the shared one, one more flush on its boot. That + // registration is made below serveEstate, whose own code did not change, + // so no declaration read here showed it, and the windows missed it until + // gitLaneRead. + codeHostRead = "v0.1.1-0.20261025200628-15c2f1a96275" + + // gitLaneRead is crewlet/crewlet#298's engine with the estate service's + // git lane, on which a node with the git role answers the git reads of + // its repositories, read where serveEstate was read again against the + // windows. The estate service registers its lanes one after another, + // each one flush, and every data node registers two, the shared lane and + // the proposals', which the windows counted as one: the proposals' lane + // came in on the branch below serveEstate, whose own code did not change, + // so the service's Serve and its lanes are read now too, and the queue's + // Serve and flush, which make each registration one flush. The git lanes + // are registered only beside a git half, which serveEstate hands the + // service only on a node with the git role, and no pod the operator runs + // has that role. Every other registration the windows count was read + // down to the queue there too: the steer answerer and the fleet broker + // register in their step's own code (armSteer, serveFleetBroker), and + // the history answerer, the inbox relay and the presence answerer each + // register one subject, in a function one call below their step + // (eventfan's Serve, (*inboxRelay).serve, presence's (*Hub).Start), + // which are read now as well, so a registration added in any of them + // moves a digest, as one added to the estate's lanes does. + gitLaneRead = "v0.1.1-0.20261026025520-adba06740326" + + // hedgeCeilingRead is crewlet/crewlet#298's engine where a leader read is + // asked again while nobody answers it and every read asked that way has + // a ceiling, read where the bounds it moved were read again against the + // windows. A leader read is asked as a log's reads were, each request + // presumed lost after its term and the next sent beside it, and one on a + // context with no deadline is held to the client's DefaultTimeout before + // it is asked ((*Leader).get, ask, askTimed, NewLeader, ReadBoundedWith), + // where get applied that bound itself: the windows' leader reads keep + // their 5 s, as does the broker's limit read, which is asked the same + // way now ((*Queue).budget). A read asked that way from the start, a + // state log's state and its record at a sequence among them, ends at its + // timing's Lookup where its caller set no deadline (ReadWith, with the + // queue's timing: (*Queue).provisioning, Clustered.Timing), where it was + // asked for as long as its caller's context lived: the windows count a + // boot's reads of a state log at that ceiling now, where they counted + // each at the term of one request for want of a bound. A certified + // listing's key index is a listing of the bucket's subjects from its + // leader, one page for a list as short as the nodes' positions + // (keysUnder, (*Leader).Subjects, page, sorted), where it was two + // requests, which takes 5 s off the join's listing of them. The nats + // bucket's lookup is held to Lookup by ReadWith rather than by a context + // of its own, its open hands the piece reads of a bucket across a link a + // longer term, and a log's read of a record across a link is given its + // crossing beside the read term, none of which moves a bound. + hedgeCeilingRead = "v0.1.1-0.20261026010533-a5bf068300dd" + + // clockSkewRead is crewlet/crewlet#298's engine that measures each node's + // clock against its broker's (clock_skew), read where the fleet's + // positions listing was read again against the windows. Each row the + // listing answers now carries the instant the broker stored it, which the + // walk of the register's node class hands on from the entry it holds + // already ((*FleetStore).eachPositionKey) and Positions sets on the row: + // the listing asks nothing it did not, so the join's listing of the + // positions keeps its 20 s. That walk was read by no row, though, and nor + // was the way from it to the certified listing it is, a fleet bucket's + // filtered walk with the leader reader it addresses + // ((*FleetStore).eachUnder, eachEntryUnder, newLeaderReader), or the + // steps of the pass between listUnder and the functions that bound them + // ((*pass).walk, stop, certify and close), or the read back of a key the + // pass did not deliver live ((*leaderReader).last, (*Leader).Last). A + // request added in any of them would have moved no digest, so they are + // read here too, and each makes no more requests than the windows count + // for a listing. No row read the way the other reads of one key the + // windows count reach that same leader read either (a log's trim floor + // and its maintenance window, the object store's fleet record and the + // rounds of its create): the store's read of a key's live value + // ((*FleetStore).get, getLatest, (*leaderReader).live) and its create + // ((*FleetStore).create). Nor did any row read what bounds the write each + // of those rounds makes before its read: a settled update, which waits + // out the leader's answer that another write to the key is still in + // flight, no longer than the client's DefaultTimeout where its caller set + // no deadline (settled, settledBucket.Update, settledBucket.settle and + // jsinflight's Decide). Each read is one leader read of the key, and each + // round's write one settled update, at the client's 5 s, and they are + // read here as well. + clockSkewRead = "v0.1.1-0.20261026011712-cbc031701107" + + // inboxLiftRead is crewlet/crewlet#298's engine that lifts a person's + // inbox out of the tracker (crewlet/crewlet@1a61828c7), read at the head + // of its branch, where the inbox relay a data node arms was read again + // against the windows. armInboxRelay hands the relay every domain whose + // applier announces a person's movements (inboxSources) and the check of + // which of them this node's copy holds out of service + // ((*localEstate).outOfService), where it handed it the tracker's copy's + // alone, and the relay delivers the movements as the inbox's own, of any + // domain (deliverInbox). Neither is a request on the boot: the relay + // still serves every domain on its node's one subject, registered once + // through (*inboxRelay).serve, whose digest holds, so a data pod's boot + // counts it as the one flush it was, and a node without data registers + // none. + inboxLiftRead = "v0.1.1-0.20261027063930-66e8ad593665" + + // brokerAuthRead is crewlet/crewlet#298's engine whose embedded broker + // admits only the fleet's keys (crewlet/crewlet@a7a499203), read at the + // head of its branch, where a node's connections to its own broker and a + // leaf's wait for the fleet were read again against the windows. Every + // connection the engine opens to its own broker is made in process, a + // clustered member's too (crewlet/crewlet@acde63711), and presents a key + // the process mints at each start ((*embeddedServer).connect). A data + // pod is a lone member and a seat pod a leaf, whose connections were in + // process already: each is still a handshake bounded by listenTimeout + // and a flush, so a pod's boot counts each connection at the 60 s it + // did. A leaf asks for the fleet's JetStream as it did, until + // clusterReadyTimeout; only its refusal moved, which now names the key + // the leaf presents, because a member that refuses that key looks from + // the leaf like one that is not there (awaitLeafReady). So a seat pod + // that every data pod refuses exits after the minute a seat pod that + // reaches none does, and no window moves. The decision that moved is + // who may link: the leaf listener admits a link only on a key derived + // from the company's keyring, which the render writes whole into every + // pod, data and seat alike, so no render changes. What it changes is + // the operator's text on what guards the leaf listener. An edit of the + // keyring that adds a key and makes it active strands no seat pod, since + // the seat pods take a new keyring only once every data pod is ready on + // it (writeSeatPool); one that removes the key they present has them + // refused between the two rolls, as the credentials guide says. + brokerAuthRead = "v0.1.1-0.20261030230815-ab6d97e67b58" + // The engine files most of the bounds on a boot are read from, and the // guide that quotes the windows they add up to. jsprovisionGo = "internal/jsprovision/jsprovision.go" + hedgeGo = "internal/jsprovision/hedge.go" + jsapiLeaderGo = "internal/jsapi/leader.go" + jsapiListingGo = "internal/jsapi/listing.go" embeddedGo = "internal/queue/jetstream/embedded.go" queueGo = "internal/queue/jetstream/queue.go" + domainLogGo = "internal/queue/jetstream/domainlog.go" stateLogGo = "internal/engine/statelog.go" backendsGo = "internal/engine/backends.go" walkGo = "internal/coord/kv/walk.go" positionsGo = "internal/coord/kv/positions.go" + fleetGo = "internal/coord/kv/fleet.go" + casLineGo = "internal/coord/kv/casline.go" + settleGo = "internal/coord/kv/settle.go" natsObjGo = "internal/objstore/natsobj/natsobj.go" troubleshootingMd = "docs/guides/troubleshooting.md" @@ -245,6 +476,17 @@ var framedKinds = []string{companyTypesGo, engineClassGo, companyExposureGo, exp // the guides that tell people what to write and what that message means. var addressCopies = []string{addressGo, companyTypesGo, engineClassGo, companyExposureGo, exposureMd, troubleshootingMd} +// boxURLCopies is every operator file that copies which addresses handed to +// sandboxes Crewlet refuses to boot on: those of addressCopies, the render +// holding a data pod back on one, the API refusing one a Company states as +// its publicBaseURL or a class's webhooks host name template makes, and the +// controller's message naming the variable and where its address came from; +// the controller's choice of which variables a pod is handed, which the +// render judges (boxURLs and receiverURL); and enginetest's BoxURLVerdicts, +// which names the addresses the boot reads for the suites that ask it. +var boxURLCopies = append([]string{"internal/controller/company_workload.go", "internal/engine/enginetest/enginetest.go"}, + addressCopies...) + // signInPosture is every operator file that copies where Crewlet lets a // second factor be optional without the acknowledgement: the render, which // holds a data pod back on it, the EngineClass API's spec.signIn, which says @@ -394,6 +636,15 @@ var bootWindows = []string{ // (API.validate, APIAuth.validate), which the contract tests hold, since // their every shape reaches the engine through it. // +// So are the addresses handed to sandboxes after them, which the engine +// refuses to boot on by a grammar of its own (ParseBoxURL and its helpers), +// and no validation reads, since they are a pod's environment: +// TestContractBoxURLShapes holds the render's verdict to the one `crewlet llm +// doctor` reports at every shape it tries, and test/crd's +// TestAddressShapesFollowTheEngine the API's on a publicBaseURL. Which +// variables the boot reads and refuses on is ReadBoxURLs' and BoxURLs', whose +// call in engine.New is not read, for engine.New's reason below. +// // The boot's bounds after them are what the controller's startup windows // add up (dataPodBootSeconds, seatPodBootSeconds and the object stores' // steps): every constant a wait is bounded by, and every function that @@ -407,7 +658,57 @@ var bootWindows = []string{ // windows leave out, past the recovery boundary: (*Engine).bootSecrets, the // boot's read of the company's secrets, which waits out a store that does not // answer for as long as the boot lives, as the pages that say what a boot -// does after /health answers 200 state. +// does after /health answers 200 state. One more is a wait the engine bounds +// by no time, which the windows count as a boot meets it: the agreement of +// the object store's fleet record, bounded by a number of attempts +// (casRetry, fleetCASAttempts), counted at the one attempt a boot meets +// unless someone removes the record meanwhile. How long a read a step makes +// may wait is read down to the function that bounds it, for every read the +// engine asks again while nobody answers it (hedge): ReadBoundedWith holds +// one whose caller set no deadline to the bound it is handed, and every one +// to its timing's Lookup. A leader read is handed its client's +// DefaultTimeout ((*Leader).get, ask, askTimed, NewLeader), as are the pages +// of a pass's key index (keysUnder, (*Leader).Subjects, page, sorted) and +// the broker's limit read ((*Queue).budget); a read of a log's state or +// record is handed Lookup itself (ReadWith, with the queue's timing, +// (*Queue).provisioning and Clustered.Timing), which a boot's read waits +// out, its context having no deadline. Each read of one key the windows +// count, of the key's live value ((*FleetStore).get, getLatest, +// (*leaderReader).live), in a round of a create ((*FleetStore).create, +// createKey) or as a certified listing's read back, is one leader read of +// the key's newest message ((*leaderReader).last, (*Leader).Last). The write +// each round of that create makes before its read is read down to its bound +// as well: it is a settled update, which waits out the leader's answer that +// another write to the key is still in flight, no longer than the client's +// DefaultTimeout where its caller set no deadline (settled, +// settledBucket.Update, settledBucket.settle, Decide). The objects bucket it +// writes is settled where the fleet store opens it (OpenFleet), which is not +// read here, since Crewlet's +// TestAConditionalWriteIsAnsweredByWhatTheLeaderDecided requires that bucket +// to be settled. So a bound moved below a step's read or write, whose own +// code need not change for it, sends somebody to the windows. And +// how many registrations a step makes is read down to the queue, for every +// step that registers what a node answers: the steer answerer and the fleet +// broker register in their step's own code (armSteer, serveFleetBroker); +// the history answerer, the inbox relay and the presence answerer each +// register one subject in the function their step calls (eventfan's Serve, +// (*inboxRelay).serve, presence's (*Hub).Start); the estate service +// registers each of its lanes in turn (estate's Serve and lanes); and every +// registration is one flush ((*Queue).Serve and flush). So a registration +// added below a step, whose own code need not change for it, sends somebody +// to the windows too. And the listing the windows count, the join's of the +// fleet's positions, is read down to the certified listing it is, and that +// listing to the functions that bound its waits: a class of the positions +// register is one walk of the register ((*FleetStore).eachPositionKey), a +// filtered walk of a fleet bucket one certified listing, its leader reader +// addressed without a request ((*FleetStore).eachUnder, eachEntryUnder, +// newLeaderReader), and a certified listing a pass begun beside its key +// index, walked to its end, stopped, certified and closed (listUnder, +// beginPass, (*pass).walk, passOver, passIdle, (*pass).stop, +// (*pass).certify, certify, (*pass).close), whose certifying read of a key +// is one leader read ((*leaderReader).last, (*Leader).Last). So a request +// added below that listing, whose own code need not change for it, sends +// somebody to the windows as well. var mirrored = []mirror{ { file: bootGo, name: "ArtifactsListener", @@ -500,8 +801,8 @@ var mirrored = []mirror{ "the catalogue's default, and refuses before a box exists what newCodingRun and sandboxCredentials " + "refuse, and a run whose environment holds a value that resolves to nothing under a key nothing " + "declares optional (runEnvironment's gap)", - digest: "b076a76f340bafe020454290babf1f76e03bb9beb78b183a720de3c07d23da75", - read: standingInstructionsRead, + digest: "8c4ebcadecdfbb3d9c29f765184ac06aa4ea1079f8c9cb3e6a8cf5708fb073e8", + read: runReposRead, }, { file: sandboxGo, name: "buildSandbox", @@ -709,8 +1010,8 @@ var mirrored = []mirror{ "tool loop (AgentModeBridgeUnsupported, NotExposed), and so is one whose CLI login cannot follow it " + "into the remote sandbox (sandboxCredentials at phase Execute); a run whose environment holds a " + "value that resolves to nothing starts without it", - digest: "05cd682090e7206bb923b20f179e3a2044d9ac2046258efb8ed921f186641032", - read: standingInstructionsRead, + digest: "4855c9c523e090c8f93dcac8905db9b1b3638955d5569fa7d5ad73337c1397e9", + read: bridgeStopRead, }, { file: agentRunGo, name: "(*agentLauncher).executorRun", @@ -884,9 +1185,10 @@ var mirrored = []mirror{ { file: mcpbridgeGo, name: "Build", against: []string{executorGo}, - decides: "a node given no CREWLET_MCP_BRIDGE_URL builds no tool bridge (the test is Configured's)", - digest: "108e637e5417c3bd768bbb6785ede3313232fb828e519f32c1a833125c37e58b", - read: bootBindRead, + decides: "a node builds no tool bridge without a base, which its boot reads from CREWLET_MCP_BRIDGE_URL " + + "through ParseBoxURL, refusing to start on a value that is set and unusable", + digest: "7ccbfd8407f92b705d9a5ff6382016aa7f1360b8c214a438fd9613709ae1f38f", + read: codeHostRead, }, { file: mcpbridgeGo, name: "Configured", @@ -998,20 +1300,91 @@ var mirrored = []mirror{ digest: "b1847a33a8f613437ac8c52830fe6abbf7840d4dd9f0ce79f0550b2c8702f8cc", read: addressShapeRead, }, + { + file: engineBoxURLGo, name: "ReadBoxURLs", + against: boxURLCopies, + decides: "a node reads the tool bridge's, the telemetry receiver's and the code host's addresses, " + + "CREWLET_MCP_BRIDGE_URL, CREWLET_SANDBOX_OTEL_RECEIVER_URL and CREWLET_BOX_GIT_URL, each through " + + "ParseBoxURL", + digest: "7b282fd465e5691eef2c90769071f45c96b6225695dc0dc6de0b4e16cf2fb834", + read: boxGitURLRead, + }, + { + file: engineBoxURLGo, name: "BoxURLs.All", + against: boxURLCopies, + decides: "the addresses a node's boot refuses on are the receiver's, the bridge's and the code host's", + digest: "3733d06f883b8ca05113b53f214d68c90d79e6f910154968e8db740b926ded9b", + read: boxGitURLRead, + }, + { + file: engineBoxURLGo, name: "BoxURLs.Refusal", + against: boxURLCopies, + decides: "a boot is refused on any of those addresses that is set and unusable, whatever the company runs", + digest: "da110da53cf23febfa9a9c8b51d4191e4a9c5050c52f37e490a7244408feb73c", + read: codeHostRead, + }, + { + file: boxURLGo, name: "ParseBoxURL", + against: boxURLCopies, + decides: "an address a sandbox dials, with the space around it dropped and unset when nothing is left, is " + + "an absolute http or https URL with no user information, a host boxHostRefusal takes, a port from 1 to " + + "65535 where it names one, and no query or fragment", + digest: "14386a77a82a7374c7e52953c60e84e5d09de8d6ce0774bf2bbb132a6ed7672e", + read: codeHostRead, + }, + { + file: boxURLGo, name: "boxHostRefusal", + against: boxURLCopies, + decides: "that host has no IPv6 zone, is UTF-8 once its escapes are decoded, holds no forbidden domain code " + + "point, and ends in a number only as an IPv4 address the WHATWG URL Standard reads", + digest: "c02b646fdb22072ec1ae077fe14ab33f4ec40eccafb45bde4965fbd546f11171", + read: codeHostRead, + }, + { + file: boxURLGo, name: "forbiddenInADomain", + against: boxURLCopies, + decides: "the forbidden domain code points are the C0 controls, space, DEL and # % / : < > ? @ [ \\ ] ^ |", + digest: "efc942efcd201819bbc4b389a782c324385ee3577f156b81a713532118ace2be", + read: codeHostRead, + }, + { + file: boxURLGo, name: "endsInANumber", + against: boxURLCopies, + decides: "a name ends in a number when its last label, one trailing dot ignored, is decimal digits or a " + + "part ipv4Number reads", + digest: "195fefaf6020f2c47e36752f235648edb47ee260937a173c43a50085427c7bdd", + read: codeHostRead, + }, + { + file: boxURLGo, name: "isIPv4Address", + against: boxURLCopies, + decides: "an IPv4 address is at most four parts ipv4Number reads, every one but the last at most 255 and the " + + "last below 256 to the power of the bytes left for it", + digest: "d9cd518b5a95dc4a62d7293975ef4c119a077054101a6049b2f321a448915465", + read: codeHostRead, + }, + { + file: boxURLGo, name: "ipv4Number", + against: boxURLCopies, + decides: "a part of an IPv4 address is hexadecimal after 0x, octal after a leading 0, and decimal otherwise", + digest: "8eb7861dd55c978ff9b4b4abd409cf7202f6beceddfa30cc1af229cf2e8276de", + read: codeHostRead, + }, { file: "internal/coord/kv/objects.go", name: "(*FleetStore).AgreeObjectBackend", against: bootWindows, - decides: "the object store's fleet record is created, or read back, in requests bounded only by the " + - "client's 5 s default", - digest: "df14c1a0c958506e2c63dc1d02480705ef8a34a0d6c0cd6e24e2406eec4f29a8", - read: bootBoundsRead, + decides: "the object store's fleet record is created, or read back, in casRetry's attempts, each a create and " + + "a leader read at the client's 5 s default, and an attempt is lost only to a record removed between the " + + "create's refusal and the read", + digest: "22378d242561f7e5c397fe5d5e9a184f947ec8144ba0908b33aa85e5d2281d95", + read: codeHostRead, }, { - file: positionsGo, name: "(*FleetStore).Floors", + file: positionsGo, name: "(*FleetStore).Floor", against: bootWindows, - decides: "the fleet's trim floors are one certified listing", - digest: "bfa507886b1a36731fb7ef23a0c0025ebd6e64a7eaebcad7d9664789cfdcd09b", - read: bootBoundsRead, + decides: "a domain's trim floor is one leader read of its key, at the client's 5 s default", + digest: "b706cb84a50391e17bad24e514ba31001e05fd7809f4bad660e945dd448b0795", + read: codeHostRead, }, { file: positionsGo, name: "(*FleetStore).Maintenance", @@ -1024,8 +1397,37 @@ var mirrored = []mirror{ file: positionsGo, name: "(*FleetStore).Positions", against: bootWindows, decides: "the fleet's positions are one certified listing", - digest: "138413da5b12c93bcc094b3e9a44623b5395b25f4c1f29d9038b0b3cd54d20b5", - read: bootBoundsRead, + digest: "88fe79922377dee667b5bc655b010eea3cf9ad394583c0c2f2fd92ddd297c87e", + read: clockSkewRead, + }, + { + file: positionsGo, name: "(*FleetStore).eachPositionKey", + against: bootWindows, + decides: "a key class of the positions register is one filtered walk of the register, each of its entries " + + "handed on with the instant the broker stored it, which the entry already carries", + digest: "ff039de2ad4de929638d75f4ed4592d3450b09c2fb6308c135615902b7ef711d", + read: clockSkewRead, + }, + { + file: fleetGo, name: "(*FleetStore).eachUnder", + against: bootWindows, + decides: "a filtered walk of a fleet bucket is one eachEntryUnder on the store's client", + digest: "24d6c44892f1ec779606075259970a7c3a60f278428c6486661c3fb1df98884f", + read: clockSkewRead, + }, + { + file: walkGo, name: "eachEntryUnder", + against: bootWindows, + decides: "a filtered walk addresses its certifying reads without a request, then is one certified listing", + digest: "42b80859a1b27b6cbdc702d5859deab28e3b5b9f744269034eab0ce203c7a1cb", + read: clockSkewRead, + }, + { + file: walkGo, name: "newLeaderReader", + against: bootWindows, + decides: "a bucket's leader reads are addressed by NewLeader for the bucket's stream, without a request", + digest: "b2fde2aad29b0b030cb4c761da926af5c2eb5c307915e46c21df67666edc1ebb", + read: clockSkewRead, }, { file: positionsGo, name: "(*FleetStore).PutAdmission", @@ -1063,6 +1465,36 @@ var mirrored = []mirror{ digest: "c35d03701efa142d2d79312757ee4923e8b028ce5b336ab3cafa30777b125f95", read: bootBoundsRead, }, + { + file: walkGo, name: "(*pass).walk", + against: bootWindows, + decides: "a pass is walked to its end by one passOver, keeping the key index when it answered during the walk", + digest: "5ee4064276f1fda1bf68996064462051a00f969c4baf4c6345e86d874ee03d06", + read: clockSkewRead, + }, + { + file: walkGo, name: "(*pass).stop", + against: bootWindows, + decides: "a pass's watcher is stopped once, by its own Stop", + digest: "bf2ef70d2391df82d61fb4a08234c45a354c82ab959dbc5a281ef27d988de124", + read: clockSkewRead, + }, + { + file: walkGo, name: "(*pass).certify", + against: bootWindows, + decides: "a pass is certified by one certify against its key index, which it waits for only when the pass " + + "ended before the index answered", + digest: "ce99a60a0adb4f6e09e5811d1a773dc2545f9d39693b5b0c3565e14cd6fd120e", + read: clockSkewRead, + }, + { + file: walkGo, name: "(*pass).close", + against: bootWindows, + decides: "a pass's close cancels its key index read and, where the pass has not read its answer, waits " + + "for it to return, asking nothing", + digest: "d167d12a0cd1285afaadea7ea94d9a456b57cfbbf908841a784c1e48d1ad197f", + read: clockSkewRead, + }, { file: backendsGo, name: "attachCoordination", against: bootWindows, @@ -1081,18 +1513,18 @@ var mirrored = []mirror{ { file: backendsGo, name: "objectsReservation", against: bootWindows, - decides: "sizing the nats bucket reads the broker's limit under LookupBudget, and carries on from the " + - "volume's free space past it", - digest: "7ba9b5772807a60484d6bbe24850daf0e6b09617a7bd9c95a28722822c38aa45", - read: bootBoundsRead, + decides: "sizing the nats bucket reads the broker's limit under LookupBudget, and carries on from a limit " + + "made of the volume's free space past it", + digest: "cfbb9b3c4231a834f970a45b379af7f50d573b5d8876dc39e936fe82f4a2864a", + read: codeHostRead, }, { file: backendsGo, name: "openObjects", against: bootWindows, decides: "the object store agrees the fleet's backend, then opens the nats bucket under a sequence ceiling" + " of its own or probes the S3 bucket", - digest: "19bd69c70076d83dff808d97561d6408d2bde7f1f1d48819f82b1ac874f28af1", - read: bootBoundsRead, + digest: "25c0df1a1666eadb808e825fddb5ac10bf2327da631b5b92070c1e7446d22945", + read: codeHostRead, }, { file: "internal/engine/core.go", name: "(*Engine).startCore", @@ -1104,16 +1536,17 @@ var mirrored = []mirror{ { file: "internal/engine/evicted.go", name: "(*stateLog).fleetGenerations", against: bootWindows, - decides: "the join's generations are one listing of the fleet's floors", - digest: "c7b4b8ded7d5a90a16ccdf3879fa9b6e74d8741208d43f845f0d5f732908ee45", - read: bootBoundsRead, + decides: "the join's generations are judged from the floors the join read, and read a log only to judge a " + + "node ahead of this one evicted, which a lone data node has none of", + digest: "0e28bf714bf8c255a8beaf4f84dc122aa95270afbf16bc81b2fc9d00dda24f3a", + read: codeHostRead, }, { file: "internal/engine/fleetbroker.go", name: "(*Engine).serveFleetBroker", against: bootWindows, decides: "a broker member registers its fleet broker answerer, one flush", - digest: "acd95e184e9549186af2396272a788ee488cf511b1ce3357fbe96d506dd28d99", - read: bootBoundsRead, + digest: "f77b2a24359ede0f1fbc0ab79168871ce395401314d9863c388ce53a61a5a5c5", + read: codeHostRead, }, { file: "internal/engine/history.go", name: "(*Engine).armHistory", @@ -1126,8 +1559,16 @@ var mirrored = []mirror{ file: "internal/engine/inboxrelay.go", name: "(*Engine).armInboxRelay", against: bootWindows, decides: "a data node registers its inbox relay, one flush", - digest: "673a18d6bcdf74dad77ace48bd70a663d2879b58bd0e5590de638434cb204d53", - read: bootBoundsRead, + digest: "f51056d70715262b7ad2a9a4d489552726857565b25405d5a28acd9cca7a43c6", + read: inboxLiftRead, + }, + { + file: "internal/engine/inboxrelay.go", name: "(*inboxRelay).serve", + against: bootWindows, + decides: "the inbox relay a data node arms (armInboxRelay) registers one subject, its node's own, through " + + "the queue's Serve", + digest: "cce917b34fdcc2b5c6f26f1b85ee9ace995b3a0b43dd159137a001a01bc7745a", + read: gitLaneRead, }, { file: "internal/engine/maintenance_mode.go", name: "(*Engine).admit", @@ -1146,9 +1587,10 @@ var mirrored = []mirror{ { file: "internal/engine/remote.go", name: "(*Engine).serveEstate", against: bootWindows, - decides: "a data node registers its estate service, one flush", - digest: "d6a146c8e4b638098a187a7b279dd60361f25c6c251590a3243bb1b1a66c720a", - read: bootBoundsRead, + decides: "a data node registers its estate service, and hands it a git half only where the node has the git " + + "role, which no pod the operator runs has", + digest: "7b2a9fbfee94698df1708f53198f434b9d3120c48a875f2b06f247a48f9e54ce", + read: gitLaneRead, }, { file: "internal/engine/remote.go", name: "(*Engine).takeCustody", @@ -1190,16 +1632,17 @@ var mirrored = []mirror{ { file: stateLogGo, name: "(*stateLog).replayable", against: bootWindows, - decides: "the join lists the fleet's positions and floors, then reads each log's stats and trim floor", - digest: "ea1c92e0718f299efaab001075f05a54cf8525842627b3f323f7191fc1b6dd0f", - read: bootBoundsRead, + decides: "the join lists the fleet's positions and reads every log's trim floor at once, then reads each " + + "log's stats and judges its floor from that one read", + digest: "8df5a23499b2d22cd15b742e83cf3f49478392e30008eed1d5d35ff129b5ed4f", + read: codeHostRead, }, { file: stateLogGo, name: "(*stateLog).start", against: bootWindows, decides: "a log's start reads its stats, attaches its consumer and verifies its checkpoint", - digest: "db62104d27fe1aba85ff011579ef018f65c2862a9b20930cf31eeed5771bcceb", - read: bootBoundsRead, + digest: "62f930ac3079f85e8704ca3ff2ff4ea7b8771af963dfd8e4c8ecaa3c723b4094", + read: codeHostRead, }, { file: stateLogGo, name: "(*stateLog).startLog", @@ -1209,11 +1652,18 @@ var mirrored = []mirror{ read: bootBoundsRead, }, { - file: stateLogGo, name: "(*stateLog).trimFloor", + file: stateLogGo, name: "(*stateLog).readFloors", against: bootWindows, - decides: "a log's trim floor is one listing of the fleet's floors", - digest: "579fb7d3962be4896893e798a1b511f22be5d4d789580da7c70ebf962ee52dc9", - read: bootBoundsRead, + decides: "the join reads the trim floor of every registered domain in one read of them all", + digest: "1acf7c934f5bf9c9fd523ca54345b6ba7a4a37fdaaf7f1db5529f82f6b49299f", + read: codeHostRead, + }, + { + file: stateLogGo, name: "readFloorsOf", + against: bootWindows, + decides: "the floors of the domains a read names are read all at once, each by its key", + digest: "7801568deb89f4c30dc12954adefd594b07127b236c4b0c599e195862b981d16", + read: codeHostRead, }, { file: "internal/engine/statelogceilings.go", name: "ceilingsFor", @@ -1233,15 +1683,46 @@ var mirrored = []mirror{ file: "internal/engine/steer.go", name: "(*Engine).armSteer", against: bootWindows, decides: "every node registers its steer answerer, one flush", - digest: "4ee600edf7605e6d684024ab1fb004dad231aab572c6f0dae31e80ad18fb07f1", - read: bootBoundsRead, + digest: "34db69352bc8adc9f9674878dad7db6d59b7000f5b2a14b5fbbb54e10cc846c4", + read: codeHostRead, + }, + { + file: "internal/estate/server.go", name: "Serve", + against: bootWindows, + decides: "the estate service registers its lanes one after the other, each through the queue's Serve, and " + + "the git lanes beside them only when it is handed a git half", + digest: "a8fb9d8a6c8a3b24aa93f81c43e81e770a9d1a15180766f72fdd183bdf1972ea", + read: gitLaneRead, + }, + { + file: "internal/estate/wire.go", name: "lanes", + against: bootWindows, + decides: "every node that serves the estate registers two lanes, the shared one and the proposals'", + digest: "90f732344404e8d6f55ea3da000d0630598a0f724861fea1e46afa81220d5e38", + read: gitLaneRead, + }, + { + file: "internal/eventfan/fanwire.go", name: "Serve", + against: bootWindows, + decides: "the history answerer a data node arms (armHistory) registers one subject, through the queue's Serve", + digest: "ca0feed0ded0e8803a7ddc32187a6f8f713460a4f2cc0d3d4a1340598f1a85a1", + read: gitLaneRead, }, { file: jsprovisionGo, name: "LookupBudget", against: bootWindows, - decides: "a lookup of a stream or bucket, and the broker's limit read, are bounded at 30 s", - digest: "fa87cb9b864068dc2bf52d7f9d17a79b008f788acbf791b1f3737e298adf9de0", - read: bootBoundsRead, + decides: "a lookup of a stream or bucket, the broker's limit read, and every read asked again while nobody " + + "answers it, a boot's read of a state log among them, are bounded at 30 s", + digest: "fa87cb9b864068dc2bf52d7f9d17a79b008f788acbf791b1f3737e298adf9de0", + read: bootBoundsRead, + }, + { + file: jsprovisionGo, name: "Clustered.Timing", + against: bootWindows, + decides: "a topology's provisioning timing takes its budgets, and bounds a read at LookupBudget whatever " + + "the topology", + digest: "1cb4c07de297eb8d8cd99cc50d7136670dc9a6b3a4599791b14a8066da4ffea3", + read: hedgeCeilingRead, }, { file: jsprovisionGo, name: "ReadBack", @@ -1321,29 +1802,45 @@ var mirrored = []mirror{ digest: "1fa472fde8f947abeb6decd310f3ef13b878eb2fd283dc54e23e8a69b3f6ddb3", read: bootBoundsRead, }, + { + file: "internal/presence/hub.go", name: "(*Hub).Start", + against: bootWindows, + decides: "the presence answerer every node starts (armPresence) registers one subject, through the queue's " + + "Serve, and runs its loops on goroutines the boot does not wait for", + digest: "73aa622c3dc1403f8811d251f7ac819722b3e785ee6da02fb68b7358a4864839", + read: gitLaneRead, + }, { file: "internal/queue/jetstream/attach.go", name: "(*Queue).attach", against: bootWindows, decides: "attaching a durable group looks up, creates and reads back its consumer, then looks it up and " + "reads it back again", - digest: "c10a5e667bc9202dd55d2322b9a559a79adfde4884020720c8f933c7e196e118", - read: bootBoundsRead, + digest: "b6a9e9419ad2f3f2f6ddbfd14c2bb8d08c8f828f4661c5d1118b53521a56cef5", + read: codeHostRead, + }, + { + file: "internal/queue/jetstream/budget.go", name: "(*Queue).budget", + against: bootWindows, + decides: "the broker's limit is a read asked again while nobody answers it, held to the client's " + + "DefaultTimeout where its caller set no deadline", + digest: "d3b790a95154bbfb5effdd81becaa86d1f61fc3378f626cacf1dba19e0526119", + read: hedgeCeilingRead, }, { file: embeddedGo, name: "(*embeddedServer).awaitLeafReady", against: bootWindows, decides: "a leaf asks for the fleet's JetStream until clusterReadyTimeout, checking the deadline only " + "before each ask", - digest: "18c689dfdc9356ecd3d8d273914a0c19d393047d2a9c69234146979fa92fadd3", - read: bootBoundsRead, + digest: "af83b256e8110005765fdda32e45006f0ff7bee4a7d2a6a1b8955c69770d9a5d", + read: brokerAuthRead, }, { file: embeddedGo, name: "(*embeddedServer).connect", against: bootWindows, decides: "a connection to the node's own broker is a handshake bounded by listenTimeout, then held to the " + "contract", - digest: "879f96f323d34070554be24a1ff2aede52334a0cb153d33147d42a1bb4c072a3", - read: bootBoundsRead, + digest: "31e5318acb11098be612c0c0fe615973a1941957318f27e859be28a110809364", + read: brokerAuthRead, }, { file: embeddedGo, name: "awaitBoot", @@ -1419,6 +1916,27 @@ var mirrored = []mirror{ digest: "050788e956291f25654a7097134e229a46c6e4cbc8d8c86d76ff23432a9a6d48", read: bootBoundsRead, }, + { + file: queueGo, name: "(*Queue).provisioning", + against: bootWindows, + decides: "the queue's reads, a log's among them, are asked on its topology's provisioning timing", + digest: "ec4ea5f0aec0a8a3b05770e91aa9ad013d80ae070cd1317d02a1863c58d65de1", + read: hedgeCeilingRead, + }, + { + file: "internal/queue/jetstream/scatter.go", name: "(*Queue).Serve", + against: bootWindows, + decides: "registering what a node answers subscribes to its subject, then flushes the subscription once", + digest: "3723b2e3ed09ba3f1a8e462b1a15351e3ecb4679adb1959a2955ee7965887980", + read: gitLaneRead, + }, + { + file: "internal/queue/jetstream/scatter.go", name: "(*Queue).flush", + against: bootWindows, + decides: "a flush waits for the broker's answer for at most flushBudget", + digest: "8fa5c9ae81c1b7fac5457e1385c4925d19030696eb6017d0d608ff275fb7fe0e", + read: gitLaneRead, + }, { file: "internal/queue/jetstream/scatter.go", name: "flushBudget", against: bootWindows, @@ -1464,11 +1982,146 @@ var mirrored = []mirror{ read: bootBoundsRead, }, { - file: "internal/jsapi/leader.go", name: "(*Leader).get", + file: fleetGo, name: "(*FleetStore).get", against: bootWindows, - decides: "a leader read on a context with no deadline is bounded by the client's DefaultTimeout", - digest: "2f238e2e50c4fd88d34c368d06b29acb40575b09da6c5e30a6cc0441ad372fe1", - read: bootBoundsRead, + decides: "the fleet store's read of one key is one getLatest on the store's client", + digest: "6b976d59a8c3cb725e30ff4cd913f2fe317c36a912e0ba804d73f8ef979d1a41", + read: clockSkewRead, + }, + { + file: walkGo, name: "getLatest", + against: bootWindows, + decides: "a read of one key addresses its leader reads without a request, then is one read of the key's live " + + "value (live)", + digest: "6e289db52cad44133ae891fd3ab23152bf91e349c8cf700ccc84cc12c3061d39", + read: clockSkewRead, + }, + { + file: walkGo, name: "(*leaderReader).live", + against: bootWindows, + decides: "a key's live value is one read of its newest message (last)", + digest: "49c30248ccfd5bdbf1fac4931841df6886e4526e165ada4155bde1d16bd0c567", + read: clockSkewRead, + }, + { + file: walkGo, name: "(*leaderReader).last", + against: bootWindows, + decides: "a key's newest message is one leader read of its subject (Last)", + digest: "1c37b5838fdc14b39f539128e1fe0c15efe5d8cf7ebe482c209b95b3f7d66b3a", + read: clockSkewRead, + }, + { + file: walkGo, name: "keysUnder", + against: bootWindows, + decides: "a pass's key index is one listing of the bucket's subjects from its leader", + digest: "390fdd0fae2295dea55d380c4e12d71fa7443c26e05626c5d77dbad61c138012", + read: hedgeCeilingRead, + }, + { + file: jsapiLeaderGo, name: "(*Leader).get", + against: bootWindows, + decides: "a leader read is asked through ask with its reader's term and bound, so one on a context with no " + + "deadline is held to the client's DefaultTimeout, and asked again inside it while nobody answers", + digest: "0cedea9143c6f0f7761ae11d2a5c25783d052005b4a52a679f40e7c1981cb08d", + read: hedgeCeilingRead, + }, + { + file: jsapiLeaderGo, name: "(*Leader).Last", + against: bootWindows, + decides: "a subject's newest message is one leader read, through get", + digest: "04a4497229210e557d517813166a097d48c082f725f297328f75798747c4a43e", + read: clockSkewRead, + }, + { + file: jsapiLeaderGo, name: "NewLeader", + against: bootWindows, + decides: "a leader reader's bound is its client's own DefaultTimeout, and its term the read term", + digest: "399059045e1b688fdca2fcceef4b843098de46a4858c328472834413f907aa7a", + read: hedgeCeilingRead, + }, + { + file: jsapiListingGo, name: "ask", + against: bootWindows, + decides: "a leader request is asked as askTimed asks it, with the term and the bound it is handed", + digest: "1ce0f6238e140b6ca0c55bdaef465adbe4f2a40b5522b41963bffd216d35c892", + read: hedgeCeilingRead, + }, + { + file: jsapiListingGo, name: "askTimed", + against: bootWindows, + decides: "a leader request is asked again at the term it is handed while nobody answers it, held to the " + + "bound it is handed (ReadBoundedWith)", + digest: "79b13c39c613883c04a1b2c775116d86c6838bf6e9b3ff624e0ad133280a6eb6", + read: hedgeCeilingRead, + }, + { + file: jsapiListingGo, name: "page", + against: bootWindows, + decides: "a page of a listing is one leader request, asked as askTimed asks it, with the term and the " + + "bound it is handed", + digest: "468cd0e20919633f78ea1061aca5adc65eb3ebb5ba0b4be95ada3228ffacca37", + read: hedgeCeilingRead, + }, + { + file: jsapiListingGo, name: "sorted", + against: bootWindows, + decides: "a listing asks no page past the first that reaches the end of its list, so a short list is one page", + digest: "13c9c190383fc71842f410477aacaee4a66d630cd215fbc0b9a7441f62f09286", + read: hedgeCeilingRead, + }, + { + file: jsapiListingGo, name: "(*Leader).Subjects", + against: bootWindows, + decides: "a listing of a stream's subjects asks its leader a page at a time, each page held to the " + + "reader's bound, the client's DefaultTimeout", + digest: "574c367ad336c710b98a82c8252b437e4f15f2ef2342f62ea26c00be5a4fee08", + read: hedgeCeilingRead, + }, + { + file: hedgeGo, name: "ReadWith", + against: bootWindows, + decides: "a read hedged from the start, a log's state and its record among them, is handed its timing's " + + "Lookup as its bound, which a boot's read waits out, its context having no deadline", + digest: "72eb5391be043593b8c06e4b59bc4b470eeff3015db304f9191c4a77de5684fc", + read: hedgeCeilingRead, + }, + { + file: hedgeGo, name: "ReadBoundedWith", + against: bootWindows, + decides: "a read asked again while nobody answers it is held to the bound it is handed where its caller set " + + "no deadline, before it is asked, and every such read to its timing's Lookup", + digest: "660ea8947b4504f311ee63fd0faaa49c6cb656b93ea3d841694be67061202613", + read: hedgeCeilingRead, + }, + { + file: hedgeGo, name: "hedge", + against: bootWindows, + decides: "a read nobody answers is asked again, every request left waiting for its answer, until one is " + + "answered or its context ends", + digest: "ea86d8197c380b8b8fbb29a107c87f21d9799573eb9762108bf8cab63c2ee34c", + read: codeHostRead, + }, + { + file: domainLogGo, name: "(*DomainLog).Stats", + against: bootWindows, + decides: "a log's stats are one read of its state, on its caller's context", + digest: "153fe4d84bad2026a4bc4ea4bfdb72ecf60a61606d4523281767a02f9b2f7e57", + read: codeHostRead, + }, + { + file: domainLogGo, name: "(*DomainLog).stateInfo", + against: bootWindows, + decides: "a log's state is read through ReadWith", + digest: "366d004324711b563b2016377bf7d1523f91474683b542a1ee79fbf9462fb73a", + read: codeHostRead, + }, + { + file: domainLogGo, name: "(*DomainLog).At", + against: bootWindows, + decides: "a log's record at a sequence is one read through ReadWith, on its caller's context, given a longer " + + "term only across a link and the same bound", + digest: "051c726fd957beac0e523ed0e0d7562e7573784dc507b96f97c6e85f38f9d255", + read: hedgeCeilingRead, }, { file: "internal/statelog/transfer.go", name: "collectOffers", @@ -1479,12 +2132,26 @@ var mirrored = []mirror{ read: bootBoundsRead, }, { - file: "internal/coord/kv/race.go", name: "(*FleetStore).retryRaces", + file: casLineGo, name: "casRetry", against: bootWindows, - decides: "a fleet write that loses a race is attempted again for as long as the client's 5 s default since" + - " the first", - digest: "5ab491aab57c57858b1b8b719c64a820599a4b5ae4a3bf1c474ba5980a5cc4b2", - read: bootBoundsRead, + decides: "a fleet write is attempted again only after it lost a race, at most fleetCASAttempts times, after a " + + "wait that grows with every attempt lost, and no time bounds the whole", + digest: "f91023d706284d3861d8e54503a9dc13fd1be098323df660d89dbe02cd10cfb1", + read: codeHostRead, + }, + { + file: casLineGo, name: "fleetCASAttempts", + against: bootWindows, + decides: "the engine's one bound on a fleet write it attempts again is a number of attempts", + digest: "b614f3a6b04f29bc7eabb16fdd28b9fe29f2dc0b0448e3f0fa47908d72b73505", + read: codeHostRead, + }, + { + file: fleetGo, name: "(*FleetStore).create", + against: bootWindows, + decides: "the fleet store's create of one key is one createKey on the store's client", + digest: "0ad4ffe5e3400dc8a40ac8751ab9748bf16880e4bb7ea58b063fcebfb7cc5bab", + read: clockSkewRead, }, { file: "internal/coord/kv/markers.go", name: "createRounds", @@ -1501,26 +2168,55 @@ var mirrored = []mirror{ digest: "d52d5def68b8d33f4b3f0e48d94386f5632a0f24fd0c5e470bd98032c32ce6c0", read: bootBoundsRead, }, + { + file: settleGo, name: "settled", + against: bootWindows, + decides: "a settled bucket's conditional writes fall back to its client's own DefaultTimeout", + digest: "d574c24c954f803deccf9622e5a8cdd97039cc6d5161e6438aa0f0e2c6c88acf", + read: clockSkewRead, + }, + { + file: settleGo, name: "settledBucket.Update", + against: bootWindows, + decides: "a settled bucket's conditional update is one settle of the client's own update", + digest: "bd17315473d3f04657610788c4961e3f029918ad04533f466534c5dab2882c90", + read: clockSkewRead, + }, + { + file: settleGo, name: "settledBucket.settle", + against: bootWindows, + decides: "a settled write is one Decide, with the bucket's timeout as its fallback", + digest: "211820629cbab521ec48f901c6a2f829b9161d761254ee384faee9f8cb3dadda", + read: clockSkewRead, + }, + { + file: "internal/jsinflight/jsinflight.go", name: "Decide", + against: bootWindows, + decides: "a publish the leader answers in flight is asked again within its caller's deadline or, where it " + + "set none, the fallback it is handed, each attempt handed that budget and no wait outlasting it", + digest: "7c7e9741b58d504b4ad9693aede7c91f9baadfd3fb96b0e5e4432fdece9805aa", + read: clockSkewRead, + }, { file: natsObjGo, name: "open", against: bootWindows, decides: "the nats bucket is looked up, created and read back, then bound, under the window it is handed", - digest: "19f98c44b5091f2ee1e02867c3c43c2fdb58d7725922de6e02fad24b4660b5b1", - read: bootBoundsRead, + digest: "e77af5243967860f1886fd619e03808868860e88c2c4a5666660a3969e85f9ee", + read: hedgeCeilingRead, }, { file: natsObjGo, name: "lookupBucket", against: bootWindows, - decides: "the bucket's lookup is bounded by the timing's Lookup", - digest: "5f6d37bb7d8d6d15f34240329eb52abdaf6fadfcbadc5c71ca953f790d6eca98", - read: bootBoundsRead, + decides: "the bucket's lookup is a read through ReadWith, bounded by the timing's Lookup", + digest: "3b0dbfbf074b7375a71a0869d86d5ee024216abc6bda6e3142d3189b577806c7", + read: hedgeCeilingRead, }, { file: natsObjGo, name: "createBucket", against: bootWindows, decides: "the bucket's create is bounded by the timing's Budget, and read back under ReadBack", - digest: "9b6c501a360dc5fa65d799d42ee922e15fe8569074a9fa0c0171b7be007cc36b", - read: bootBoundsRead, + digest: "684a164cd0e747e24c77ad15708c7b266a66cae0994c3eefb6e8ef626798ed35", + read: codeHostRead, }, { file: natsObjGo, name: "boundBucket", diff --git a/internal/render/optional_test.go b/internal/render/optional_test.go index 0a6e111..300064b 100644 --- a/internal/render/optional_test.go +++ b/internal/render/optional_test.go @@ -540,7 +540,7 @@ func TestCodeWorkGoesWithoutAMissingOptionalSignIn(t *testing.T) { fixer.Spec.Sandbox.OptionalKeys = append(fixer.Spec.Sandbox.OptionalKeys, envClaudeSignIn) } if c.step { - step := v1alpha1.SandboxSetupStep{Name: "sign-in", Env: []v1alpha1.SecretEnvVar{{ + step := v1alpha1.SandboxSetupStep{Name: "sign-in", Env: []v1alpha1.SandboxEnvVar{{ Name: envClaudeSignIn, ValueFrom: v1alpha1.SecretValueSource{SecretKeyRef: stepSignIn}, }}} if c.stepOptional { diff --git a/internal/render/org.go b/internal/render/org.go index f49da7c..9019d72 100644 --- a/internal/render/org.go +++ b/internal/render/org.go @@ -326,6 +326,9 @@ func (r *renderer) role(at Path, s *v1alpha1.Seat) roleDoc { } } } + if access := spec.Code.Access; access != "" { + doc.Code = &roleCodeDoc{Access: codeAccessTier(access)} + } doc.Integrations = r.roleIntegrations(s) doc.Schedules = r.schedules(at, r.seatSchedules[s.Name]) return doc @@ -423,11 +426,8 @@ func (r *renderer) roleIntegrations(s *v1alpha1.Seat) roleIntegrationsDoc { SigningSecret: r.secret(creds, v1alpha1.SlackSigningSecretKey), } } - if g := in.GitHub; g.Tier != "" || len(g.Repos) > 0 || g.App.ID != 0 { - doc.GitHub = &roleGitHubDoc{ - Tier: gitHubAccessTier(g.Tier), - Repos: g.Repos, - } + if g := in.GitHub; len(g.Repos) > 0 || g.App.ID != 0 { + doc.GitHub = &roleGitHubDoc{Repos: g.Repos} if app := g.App; app.ID != 0 { doc.GitHub.AppID = app.ID doc.GitHub.AppSlug = app.Slug diff --git a/internal/render/streamfloor_test.go b/internal/render/streamfloor_test.go index c9848eb..28381eb 100644 --- a/internal/render/streamfloor_test.go +++ b/internal/render/streamfloor_test.go @@ -15,14 +15,16 @@ import ( ) // What a fresh node of Crewlet reserves on its event stream before it boots -// (its engine.BrokerFloorFor): its seven state logs' ceilings at their -// floors, and, with the objects in the stream's own bucket, that bucket -// beside them at a quarter of the stream. TestContractStreamFloor holds both -// to the figure `crewlet validate` names, and TestStreamFloorFiguresAreTheEngines -// holds every hand-written comment and page that quotes one to these. +// (its engine.BrokerFloorFor): its eight state logs' ceilings at their +// floors, the code host's log among them, which every node reserves whether +// or not the company hosts code, and, with the objects in the stream's own +// bucket, that bucket beside them at a quarter of the stream. +// TestContractStreamFloor holds both to the figure `crewlet validate` names, +// and TestStreamFloorFiguresAreTheEngines holds every hand-written comment +// and page that quotes one to these. const ( - freshNodeLogsFloor int64 = 6979321856 - freshNodeBucketFloor int64 = 9305762474 + freshNodeLogsFloor int64 = 8053063680 + freshNodeBucketFloor int64 = 10737418239 ) // The EngineClass rules that floor the data volume, as controller-gen writes diff --git a/internal/render/testdata/company/code-access.input.yaml b/internal/render/testdata/company/code-access.input.yaml new file mode 100644 index 0000000..47e906f --- /dev/null +++ b/internal/render/testdata/company/code-access.input.yaml @@ -0,0 +1,37 @@ +# An agent seat with a code access tier and nothing it applies to: no GitHub +# Integration in the company, and no GitHub identity on the seat. Crewlet +# runs the seat, and warns that the tier grants nothing. +apiVersion: crewlet.ai/v1alpha1 +kind: Company +metadata: + name: shed +spec: + engineClassName: basic + companyName: Shed + mission: Keep the build scripts tidy. +--- +apiVersion: crewlet.ai/v1alpha1 +kind: LLMProvider +metadata: + name: claude +spec: + companyRef: + name: shed + agent: ClaudeCode + model: claude-sonnet-5 + auth: + mode: APIKey + apiKeySecretRef: + name: anthropic + key: apiKey +--- +apiVersion: crewlet.ai/v1alpha1 +kind: Seat +metadata: + name: tidier +spec: + companyRef: + name: shed + goal: Tidy the build scripts. + code: + access: Review diff --git a/internal/render/testdata/company/code-access.yaml b/internal/render/testdata/company/code-access.yaml new file mode 100644 index 0000000..bd512d9 --- /dev/null +++ b/internal/render/testdata/company/code-access.yaml @@ -0,0 +1,23 @@ +mission: Keep the build scripts tidy. +name: Shed +providers: + llm: + claude: + agent: claude-code + auth: + api_key: ${K8S_ANTHROPIC_API_KEY_NP33B5LRRKLS} + mode: api-key + model: claude-sonnet-5 + state_dir: /var/lib/crewlet/llm-cli/claude + llm_order: + - claude +roles: +- code: + access: review + goal: Tidy the build scripts. + handle: tidier + kind: agent + name: tidier + placement: + labels: + crewlet.ai/pool: data diff --git a/internal/render/testdata/company/github-enterprise.input.yaml b/internal/render/testdata/company/github-enterprise.input.yaml index 746affa..14b77a4 100644 --- a/internal/render/testdata/company/github-enterprise.input.yaml +++ b/internal/render/testdata/company/github-enterprise.input.yaml @@ -37,9 +37,10 @@ metadata: spec: companyRef: name: forge + code: + access: FullAccess integrations: github: - tier: FullAccess app: id: 42 slug: forge-builder @@ -54,8 +55,9 @@ metadata: spec: companyRef: name: forge + code: + access: ReadOnly integrations: github: - tier: ReadOnly repos: - forge/api diff --git a/internal/render/testdata/company/github-enterprise.yaml b/internal/render/testdata/company/github-enterprise.yaml index c3ffd4e..cfe2643 100644 --- a/internal/render/testdata/company/github-enterprise.yaml +++ b/internal/render/testdata/company/github-enterprise.yaml @@ -12,26 +12,28 @@ integrations: webhook_secret: ${K8S_FORGE_GITHUB_WEBHOOK_SECRET_TBDDG2MGKIUZ} name: Forge roles: -- handle: builder +- code: + access: full_access + handle: builder integrations: github: app_id: 42 app_slug: forge-builder installation_id: 4242 private_key: ${K8S_BUILDER_GITHUB_APP_PRIVATE_KEY_XJ3ZLBDFZG45} - tier: full_access webhook_secret: ${K8S_BUILDER_GITHUB_APP_WEBHOOK_SECRET_2FHJJ6YLPUEG} kind: agent name: builder placement: labels: crewlet.ai/pool: data -- handle: reviewer +- code: + access: read_only + handle: reviewer integrations: github: repos: - forge/api - tier: read_only kind: agent name: reviewer placement: diff --git a/internal/render/testdata/company/nimbus.yaml b/internal/render/testdata/company/nimbus.yaml index bc0fba9..8fbb5da 100644 --- a/internal/render/testdata/company/nimbus.yaml +++ b/internal/render/testdata/company/nimbus.yaml @@ -114,6 +114,8 @@ units: roles: - behavioral_guidelines: - Prefer small pull requests. + code: + access: review email: ana@nimbus.example goal: Keep CI green and deploys boring. handle: ana @@ -125,7 +127,6 @@ units: private_key: ${K8S_ANA_GITHUB_APP_PRIVATE_KEY_JSUSQ7FYDCSN} repos: - nimbus-hq/nimbuscore - tier: review webhook_secret: ${K8S_ANA_GITHUB_APP_WEBHOOK_SECRET_DE4RFGLQA4OW} slack: bot_token: ${K8S_ANA_SLACK_BOT_TOKEN_HGFIPUYDX4KF} diff --git a/internal/render/testdata/company/org-chart.input.yaml b/internal/render/testdata/company/org-chart.input.yaml index 434b957..962b3d8 100644 --- a/internal/render/testdata/company/org-chart.input.yaml +++ b/internal/render/testdata/company/org-chart.input.yaml @@ -16,14 +16,14 @@ spec: timeZone: America/Phoenix tracker: backend: Worklet - worklet: - inboxRetentionDays: 90 knowledge: backend: Doclet scope: [ENG, OPS] skillsContainer: "" rootSpace: ACME vectors: false + inbox: + retentionSeconds: 7776000 tokenBudget: day: 1000000 week: 5000000 diff --git a/internal/render/testdata/company/org-chart.yaml b/internal/render/testdata/company/org-chart.yaml index 3bad1be..5b202e0 100644 --- a/internal/render/testdata/company/org-chart.yaml +++ b/internal/render/testdata/company/org-chart.yaml @@ -1,3 +1,5 @@ +inbox: + retention_days: 90 integrations: check_interval_seconds: 300 knowledge: @@ -71,8 +73,6 @@ token_budget: week: 5000000 tracker: backend: worklet - worklet: - inbox_retention_days: 90 turn_engine: extension_enabled: false max_iterations: 4 diff --git a/internal/render/testdata/company/secret-positions.input.yaml b/internal/render/testdata/company/secret-positions.input.yaml index 5f316f0..ba52704 100644 --- a/internal/render/testdata/company/secret-positions.input.yaml +++ b/internal/render/testdata/company/secret-positions.input.yaml @@ -390,6 +390,8 @@ spec: secretKeyRef: name: coder-sandbox key: registryToken + code: + access: Review integrations: slack: credentialsSecretRef: @@ -400,7 +402,6 @@ spec: name: coder-mattermost key: botToken github: - tier: Review app: id: 7 slug: vault-coder diff --git a/internal/render/testdata/company/secret-positions.yaml b/internal/render/testdata/company/secret-positions.yaml index 7612054..7d978de 100644 --- a/internal/render/testdata/company/secret-positions.yaml +++ b/internal/render/testdata/company/secret-positions.yaml @@ -138,14 +138,15 @@ units: LINEAR_TEAM_TOKEN: ${K8S_ENGINEERING_LINEAR_TOKEN_GQ3JHNIKTWHB} name: engineering roles: - - handle: coder + - code: + access: review + handle: coder integrations: github: app_id: 7 app_slug: vault-coder installation_id: 77 private_key: ${K8S_CODER_GITHUB_APP_PRIVATE_KEY_MBKQYEC7LV3R} - tier: review webhook_secret: ${K8S_CODER_GITHUB_APP_WEBHOOK_SECRET_PS7VGCGUUI6G} mattermost: bot_token: ${K8S_CODER_MATTERMOST_BOT_TOKEN_TEAYYSXE5AJ7} diff --git a/test/crd/addresses_engine_test.go b/test/crd/addresses_engine_test.go index 5d50d0a..683ec49 100644 --- a/test/crd/addresses_engine_test.go +++ b/test/crd/addresses_engine_test.go @@ -23,6 +23,7 @@ import ( v1 "github.com/crewlet/crewlet-operator/api/v1alpha1" "github.com/crewlet/crewlet-operator/internal/engine/enginetest" "github.com/crewlet/crewlet-operator/internal/render" + "github.com/crewlet/crewlet-operator/internal/secretsync" ) // baseURLShapes are the addresses TestAddressShapesFollowTheEngine offers as @@ -87,19 +88,36 @@ var apiRefusesMore = map[string]string{ "http://h.example ": "white space around the address, which the engine trims and the API refuses as no URL", } -// zoneBracket is why the API refuses as a publicBaseURL an IPv6 address -// whose zone holds a ']', which Crewlet takes there. -const zoneBracket = "a ']' after an IPv6 zone: Go's URL parser, the engine's as well as the API server's, ends the " + - "bracketed address at the last ']' and reads the rest into the zone, so the engine starts on a host no vendor " + - "or sandbox can reach; the API takes an IPv6 address in brackets whose zone holds no ']'" +// shortIPv4 is why the API refuses as a publicBaseURL an IPv4 address in a +// form other than four decimal octets, which Crewlet takes there, and the URL +// parser of a sandbox it is handed to reads as an address. +const shortIPv4 = "a host whose last label is a number and that is not four decimal octets: a sandbox's URL parser " + + "reads it as an IPv4 address in a short form, and a vendor's may read it as a name, so the API takes an IPv4 " + + "address written as four decimal octets alone, as for a dashboardBaseURL" + +// escapedName is why the API refuses as a publicBaseURL a host name with a +// percent-encoded character, which Crewlet takes there and a sandbox's URL +// parser decodes. +const escapedName = "a percent-encoded host name: the API refuses every '%' in a host, the IPv6 zone and the " + + "escapes a sandbox's URL parser refuses among them, so a host is written as it is" // publicRefusesMore are the shapes the API refuses as a publicBaseURL alone -// although Crewlet takes them there, and why: as a dashboardBaseURL Crewlet -// refuses every IPv6 zone. +// although Crewlet takes them there, as api.public.external_url and as an +// address it hands a sandbox, and why. var publicRefusesMore = map[string]string{ - "http://[fe80::1%25eth0]]/": zoneBracket, - "http://[fe80::1%25eth0]h]/p": zoneBracket, - "http://[fe80::1%25eth0]fe80::1]": zoneBracket, + "http://127.1:8080": shortIPv4, + "http://127.1": shortIPv4, + "http://2130706433": shortIPv4, + "http://0x7f.0.0.1": shortIPv4, + "http://0177.0.0.1": shortIPv4, + "http://192.168.010.1": shortIPv4, + "http://1.2.3.4.": shortIPv4, + "http://01.2.3.4": shortIPv4, + "http://1.2.3.04": shortIPv4, + "http://1.2.3": shortIPv4, + "http://123.": shortIPv4, + "http://0x.": shortIPv4, + "http://b%C3%BCcher.example": escapedName, } // apiHostShapes are the host name templates TestAddressShapesFollowTheEngine @@ -118,7 +136,10 @@ var apiHostShapes = []string{ // suite runs, both ways where that is honest: every shape the API admits // renders to a crewlet.yaml `crewlet validate` takes, so no Company the API // admits crash-loops its data pod; and every shape it refuses Crewlet refuses -// too, but for apiRefusesMore and publicRefusesMore. A Company's two +// too, but for apiRefusesMore and publicRefusesMore. The public address is +// also the one the operator hands sandboxes, as the tool bridge's and the +// telemetry receiver's, so it is held to the engine's verdict on it there +// too, which `crewlet llm doctor` reports. A Company's two // addresses are offered to both API servers, the one whose URL parser takes // a second ':' after a host too (looseURLParser): a host and port the engine // refuses are refused by the CRD's own rules on either, whatever isURL() and @@ -337,12 +358,22 @@ func expectLooseParser(t *testing.T, namespace string) { func holdAddress(t *testing.T, shape, value string, admissions []admission, field, address string) { t.Helper() engine := engineRefuses(t, field, address) + // The public address is handed to sandboxes too, and refused there by a + // grammar of the engine's own. + box := field == render.FieldPublicURL && !engine && boxURLRefused(t, address) + engine = engine || box why, more := apiRefusesMore[shape] + if more && box { + // apiRefusesMore's reasons hold against Crewlet's Tier A: as an + // address handed to sandboxes it refuses a user, a query and a + // fragment too, so the refusal is not the API's alone there. + more = false + } if public, ok := publicRefusesMore[shape]; ok && field == render.FieldPublicURL { why, more = public, true } if more && engine { - t.Errorf("%q is listed as refused by the API alone (%s), and crewlet validate refuses %s %q", + t.Errorf("%q is listed as refused by the API alone (%s), and Crewlet refuses %s %q", shape, why, field, address) } rendered := false @@ -353,7 +384,7 @@ func holdAddress(t *testing.T, shape, value string, admissions []admission, fiel t.Errorf("%q is listed as refused by the API alone (%s), and %s admits %q", shape, why, v.server, value) } case v.admitted && engine: - t.Errorf("%s admits %q, and crewlet validate refuses %s %q, so the data pod would crash-loop", + t.Errorf("%s admits %q, and Crewlet refuses %s %q, so the data pod would crash-loop", v.server, value, field, address) case v.admitted && !rendered: rendered = true @@ -368,11 +399,47 @@ func holdAddress(t *testing.T, shape, value string, admissions []admission, fiel "so the data pod would crash-loop:\n%s", v.server, value, field, address, out) } case !v.admitted && !engine: - t.Errorf("%s refuses %q, and crewlet validate takes %s %q", v.server, value, field, address) + t.Errorf("%s refuses %q, and Crewlet takes %s %q", v.server, value, field, address) } } } +// boxURLRefused is whether Crewlet refuses to boot with address as one it +// hands its sandboxes, as the tool bridge's and the telemetry receiver's +// alike, which `crewlet llm doctor` reports for a company with a provider; +// it fails the test when the two verdicts differ, since the engine reads both +// by one grammar. +func boxURLRefused(t *testing.T, address string) bool { + t.Helper() + provider := llmProvider("probe", "claude") + in := &render.Input{ + Company: company("probe", companyName), + Class: engineClass("", "basic"), + LLMProviders: []v1.LLMProvider{*provider}, + StoreName: secretsync.StoreName, + ProvisionedName: secretsync.ProvisionedName, + } + in.Company.UID = types.UID("00000000-0000-4000-8000-000000000002") + in.Company.Spec.EngineClassName = in.Class.Name + s := seat("probe", "prober") + s.Spec.LLM.Default = []v1.LLMProviderReference{{Name: provider.Name}} + in.Seats = []v1.Seat{*s} + doc, err := render.Company(in) + if err != nil { + t.Fatalf("rendering the company the engine is asked about: %v", err) + } + env := map[string]string{render.EnvBridgeURL: address, render.EnvReceiverURL: address} + for _, k := range doc.Refs { + env[secretsync.StoreName(k)] = "probe-" + k.Name + "-" + k.Key + } + refused := enginetest.BoxURLVerdicts(t, doc.YAML, env, render.EnvBridgeURL, render.EnvReceiverURL) + if refused[render.EnvBridgeURL] != refused[render.EnvReceiverURL] { + t.Fatalf("Crewlet refuses %q as %s: %v, and as %s: %v", address, + render.EnvBridgeURL, refused[render.EnvBridgeURL], render.EnvReceiverURL, refused[render.EnvReceiverURL]) + } + return refused[render.EnvBridgeURL] +} + // bootstrapInput is a data pod's Tier A for a Company of namespace tenant-a, // on the given addresses. func bootstrapInput(external, public string, origins []string) *render.BootstrapInput { @@ -406,7 +473,12 @@ func inputWith(field, address string) *render.BootstrapInput { case render.FieldExternalURL: in.ExternalURL = address case render.FieldPublicURL: + // The public address is the one the operator hands sandboxes too, + // where the company has them dial the engine back. in.PublicURL = address + in.BoxURLs = []render.BoxURL{ + {Variable: render.EnvBridgeURL, URL: address}, {Variable: render.EnvReceiverURL, URL: address}, + } case render.FieldAllowedOrigins: in.AllowedOrigins = []string{address} case render.FieldArtifactsURL: diff --git a/test/crd/company_test.go b/test/crd/company_test.go index d0e1070..480b8d9 100644 --- a/test/crd/company_test.go +++ b/test/crd/company_test.go @@ -6,6 +6,8 @@ import ( "strings" "testing" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/client" v1 "github.com/crewlet/crewlet-operator/api/v1alpha1" @@ -14,15 +16,29 @@ import ( // platformURLVar is a variable a platform sets on the engine. const platformURLVar = "PLATFORM_URL" +// secondsPerDay is a day in seconds: the inbox's horizon is whole days, +// stated in seconds. +const secondsPerDay = 24 * 60 * 60 + func TestCompanyDefaults(t *testing.T) { c := company(newNamespace(t), companyName) // A backend is defaulted inside a tracker or knowledge block that is // there; an absent block is left to Crewlet, which derives its own. So - // each block states one other setting, and the worklet one is admitted - // only because its backend defaults to Worklet before the rule runs. - c.Spec.Tracker.Worklet.InboxRetentionDays = 90 + // the knowledge block states one other setting, and the tracker block, + // which holds nothing but its backend, is sent empty: only an + // unstructured object can send it, since the typed client drops an empty + // block. c.Spec.Knowledge.Scope = []string{containerKey} - if err := k8sClient.Create(ctx, c); err != nil { + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(c) + if err != nil { + t.Fatal(err) + } + obj := &unstructured.Unstructured{Object: content} + obj.SetGroupVersionKind(v1.GroupVersion.WithKind("Company")) + if err := unstructured.SetNestedMap(obj.Object, map[string]any{}, "spec", "tracker"); err != nil { + t.Fatal(err) + } + if err := k8sClient.Create(ctx, obj); err != nil { t.Fatal(err) } got := &v1.Company{} @@ -52,10 +68,7 @@ func fullCompany(c *v1.Company) { c.Spec.EngineClassName = "standard" c.Spec.Mission = "Ship a reliable cloud storage product." c.Spec.TimeZone = "America/Argentina/Buenos_Aires" - c.Spec.Tracker = v1.TrackerSettings{ - Backend: v1.TrackerWorklet, - Worklet: v1.WorkletSettings{InboxRetentionDays: 90}, - } + c.Spec.Tracker = v1.TrackerSettings{Backend: v1.TrackerWorklet} c.Spec.Knowledge = v1.KnowledgeSettings{ Backend: v1.KnowledgeDoclet, Scope: []string{containerKey, "ENG2"}, @@ -63,6 +76,7 @@ func fullCompany(c *v1.Company) { RootSpace: "HQ", Vectors: new(true), } + c.Spec.Inbox = v1.InboxSettings{RetentionSeconds: 90 * secondsPerDay} c.Spec.TokenBudget = v1.TokenBudget{Day: 2_000_000} c.Spec.LLM.Order = []v1.LLMProviderReference{ {Name: providerName}, @@ -104,6 +118,9 @@ func TestCompanyValidation(t *testing.T) { public := func(address string) func(*v1.Company) { return func(c *v1.Company) { c.Spec.Exposure.PublicBaseURL = address } } + inbox := func(seconds int32) func(*v1.Company) { + return func(c *v1.Company) { c.Spec.Inbox.RetentionSeconds = seconds } + } runCreateCases(t, company, []createCase[*v1.Company]{ {name: "minimal company is accepted"}, @@ -123,14 +140,34 @@ func TestCompanyValidation(t *testing.T) { want: msgThreeReplicas, }, { - name: "a worklet block on backend None is refused", + // The horizon ages every notice in a person's inbox, whichever + // part of the engine wrote it, so it is the company's whatever + // tracker the company runs, or none. + name: "an inbox horizon on a company with no tracker is accepted", mutate: func(c *v1.Company) { - c.Spec.Tracker = v1.TrackerSettings{ - Backend: v1.TrackerNone, - Worklet: v1.WorkletSettings{InboxRetentionDays: 90}, - } + c.Spec.Tracker.Backend = v1.TrackerNone + c.Spec.Inbox.RetentionSeconds = 90 * secondsPerDay }, - want: "worklet applies to backend Worklet only", + }, + // Crewlet's bounds: below a month an inbox stops being one, and past + // ten years it is a second copy of the history it points at. + {name: "an inbox horizon of a month is accepted", mutate: inbox(30 * secondsPerDay)}, + {name: "an inbox horizon of ten years is accepted", mutate: inbox(3650 * secondsPerDay)}, + { + name: "an inbox horizon under a month is refused", + mutate: inbox(29 * secondsPerDay), + want: "spec.inbox.retentionSeconds in body should be greater than or equal to 2592000", + }, + { + name: "an inbox horizon over ten years is refused", + mutate: inbox(3651 * secondsPerDay), + want: "spec.inbox.retentionSeconds in body should be less than or equal to 315360000", + }, + { + // Within the bounds, but Crewlet counts the horizon in days. + name: "an inbox horizon of 90 days and an hour is refused", + mutate: inbox(90*secondsPerDay + 60*60), + want: "retentionSeconds must be whole days, a multiple of 86400", }, { // The second reference names the same provider: kind defaults @@ -481,18 +518,47 @@ func TestCompanyValidation(t *testing.T) { mutate: public("https://[fe80::1%25]:8443"), want: msgPublicBaseURLHost, }, + // The public address is handed to sandboxes, whose URL parser + // refuses what Crewlet then refuses to boot on. { - // No browser reaches the public address, and Crewlet takes a - // zone there. - name: "a publicBaseURL with an IPv6 zone is accepted", + name: "a publicBaseURL with an IPv6 zone is refused", mutate: public("https://[fe80::1%25eth0]:8443"), + want: msgPublicBaseURLPercent, + }, + { + name: "a publicBaseURL with an escaped '%' in its host name is refused", + mutate: public("https://hooks%25nimbus.example"), + want: msgPublicBaseURLPercent, + }, + { + name: "a publicBaseURL holding an unfilled placeholder is refused", + mutate: public("https://:8443"), + want: msgPublicBaseURLName, + }, + { + name: "a publicBaseURL with a ']' in its host name is refused", + mutate: public("https://hooks]nimbus.example"), + want: msgPublicBaseURLName, + }, + { + name: "a publicBaseURL whose host name ends in a number is refused", + mutate: public("https://hooks.nimbus.123"), + want: msgPublicBaseURLNumber, + }, + { + name: "a publicBaseURL whose host name ends in a hexadecimal number is refused", + mutate: public("https://hooks.nimbus.0x1F:8443"), + want: msgPublicBaseURLNumber, }, { - // No browser compares an origin with the public address, so - // Crewlet takes a host there that it refuses as a browser's. - name: "a publicBaseURL of an IPv4 short form is accepted", + name: "a publicBaseURL of an IPv4 short form is refused", mutate: public("https://127.1"), + want: msgPublicBaseURLNumber, }, + {name: "a publicBaseURL of an IPv4 address is accepted", mutate: public("https://192.0.2.10:8443")}, + {name: "a publicBaseURL of an IPv6 address is accepted", mutate: public("https://[2001:db8::1]:8443")}, + {name: "a publicBaseURL whose last label holds a letter is accepted", mutate: public("https://hooks.nimbus.1x")}, + {name: "a publicBaseURL with a ']' in its path is accepted", mutate: public("https://hooks.nimbus.example/a]b")}, { name: "a publicBaseURL with a path is accepted", mutate: func(c *v1.Company) { c.Spec.Exposure.PublicBaseURL = "https://relay.nimbus.example/nimbus/" }, @@ -654,11 +720,14 @@ func TestCompanyTopology(t *testing.T) { func TestCompanyEngineConfigReservedKeys(t *testing.T) { reserved := []string{ - "name", "mission", "vision", "policies", "timezone", "tracker", "knowledge", + "name", "mission", "vision", "policies", "timezone", "tracker", "knowledge", "inbox", "token_budget", "providers", "integrations", "mcp_servers", "roles", "units", } - const msgReservedKey = "engineConfig may not set name, mission" - cases := make([]createCase[*v1.Company], 0, len(reserved)+4) + const ( + msgReservedKey = "engineConfig may not set name, mission" + msgCodeHost = "engineConfig may not set code: Crewlet's code host is not offered in v1alpha1" + ) + cases := make([]createCase[*v1.Company], 0, len(reserved)+6) cases = append(cases, createCase[*v1.Company]{ name: "an unreserved key is accepted", mutate: engineConfig("learning", `{"enabled":false}`), }) @@ -686,6 +755,20 @@ func TestCompanyEngineConfigReservedKeys(t *testing.T) { mutate: engineConfig("Turn-Engine", `{}`), want: "spec.engineConfig[0].name", }, + // No pod the operator runs serves git, so the code host is held off + // until the API offers it, whatever the block holds. + createCase[*v1.Company]{ + name: "the code host is refused", + mutate: engineConfig("code", `{"native":true,"repos":[{"name":"web","owner":{"unit":"engineering"}}]}`), + want: msgCodeHost, + }, + createCase[*v1.Company]{ + name: "the code host is refused beside a key that is accepted", + mutate: func(c *v1.Company) { + c.Spec.EngineConfig = append(engineConfigOf("learning"), engineConfigOf("code")...) + }, + want: msgCodeHost, + }, ) runCreateCases(t, company, cases) } diff --git a/test/crd/engineclass_test.go b/test/crd/engineclass_test.go index 911fba8..024aed3 100644 --- a/test/crd/engineclass_test.go +++ b/test/crd/engineclass_test.go @@ -1,12 +1,17 @@ package crd_test import ( + "regexp" "strings" "testing" + "time" corev1 "k8s.io/api/core/v1" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/util/retry" "sigs.k8s.io/controller-runtime/pkg/client" v1 "github.com/crewlet/crewlet-operator/api/v1alpha1" @@ -81,31 +86,31 @@ func TestEngineClassObjects(t *testing.T) { }, { name: "a storage size just under the floor with files in the stream's bucket is refused", - mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("12287Mi")) }, + mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("14335Mi")) }, want: msgStorageFloorNATS, }, { - name: "a 9Gi volume with files in the stream's bucket, the default, is refused", - mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("9Gi")) }, + name: "a 10Gi volume with files in the stream's bucket, the default, is refused", + mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("10Gi")) }, want: msgStorageFloorNATS, }, { - name: "a 9Gi volume with files in the stream's bucket, named, is refused", + name: "a 10Gi volume with files in the stream's bucket, named, is refused", mutate: func(c *v1.EngineClass) { - c.Spec.Data.Storage.Size = new(resource.MustParse("9Gi")) + c.Spec.Data.Storage.Size = new(resource.MustParse("10Gi")) c.Spec.Objects = v1.ObjectSettings{Backend: v1.ObjectBackendNATS} }, want: msgStorageFloorNATS, }, { name: "a storage size of exactly the floor with files in the stream's bucket is accepted", - mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("12Gi")) }, + mutate: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("14Gi")) }, }, { name: "a storage size just under the floor with files in S3 is refused", mutate: func(c *v1.EngineClass) { s3("")(c) - c.Spec.Data.Storage.Size = new(resource.MustParse("9215Mi")) + c.Spec.Data.Storage.Size = new(resource.MustParse("10239Mi")) }, want: msgStorageFloor, }, @@ -113,7 +118,7 @@ func TestEngineClassObjects(t *testing.T) { name: "a storage size of exactly the floor with files in S3 is accepted", mutate: func(c *v1.EngineClass) { s3("")(c) - c.Spec.Data.Storage.Size = new(resource.MustParse("9Gi")) + c.Spec.Data.Storage.Size = new(resource.MustParse("10Gi")) }, }, { @@ -591,10 +596,25 @@ func TestEngineClassExposure(t *testing.T) { name: "an API hostname ending in a variable is accepted", mutate: apiGatewayHost("{{uid}}.{{name}}.{{namespace}}"), }, + // The webhooks' host is no browser's address, but the engine hands + // it to sandboxes, whose URL parser reads a host whose last label is + // a number as an IPv4 address. { - // The webhooks' host is no browser's address. - name: "a webhooks hostname ending in a number is accepted", + name: "a webhooks hostname ending in a number is refused", mutate: gatewayHost("{{uid}}.hooks.123"), + want: msgHostnameNumber, + }, + { + name: "a webhooks hostname ending in a hexadecimal number is refused", + mutate: gatewayHost("{{uid}}.hooks.0x1f"), + want: msgHostnameNumber, + }, + {name: "a webhooks hostname ending in a label with a letter is accepted", mutate: gatewayHost("{{uid}}.hooks.123x")}, + { + // Whether {{namespace}} is a number is the Company's to say, so + // the operator decides it per Company (AddressRefused). + name: "a webhooks hostname ending in a variable is accepted", + mutate: gatewayHost("{{uid}}.{{name}}.{{namespace}}"), }, // Under networkPolicy Isolated the operator has to know where an @@ -1123,15 +1143,15 @@ func TestEngineClassStartupWindow(t *testing.T) { } } runCreateCases(t, engineClass, []createCase[*v1.EngineClass]{ - {name: "the boot's own bounds are accepted", mutate: window(1636)}, + {name: "the boot's own bounds are accepted", mutate: window(1461)}, {name: "a day is accepted", mutate: window(86400)}, { - name: "under the boot's own bounds is refused", mutate: window(1635), - want: "should be greater than or equal to 1636", + name: "under the boot's own bounds is refused", mutate: window(1460), + want: "should be greater than or equal to 1461", }, {name: "over a day is refused", mutate: window(86401), want: "should be less than or equal to 86400"}, - {name: "the boot's own bounds with objects in S3 are accepted", mutate: inS3(1776)}, - {name: "under them with objects in S3 is refused", mutate: inS3(1775), want: msgStartupFloorS3}, + {name: "the boot's own bounds with objects in S3 are accepted", mutate: inS3(1601)}, + {name: "under them with objects in S3 is refused", mutate: inS3(1600), want: msgStartupFloorS3}, {name: "objects in S3 and no window is accepted", mutate: inS3(0)}, }) } @@ -1142,8 +1162,8 @@ func TestEngineClassStartupWindow(t *testing.T) { func TestEngineClassStartupWindowFollowsTheObjects(t *testing.T) { runUpdateCases(t, engineClass, []updateCase[*v1.EngineClass]{ { - name: "a class at 1636 seconds moving its files to S3 is refused", - before: func(c *v1.EngineClass) { c.Spec.Data.StartupSeconds = 1636 }, + name: "a class at 1461 seconds moving its files to S3 is refused", + before: func(c *v1.EngineClass) { c.Spec.Data.StartupSeconds = 1461 }, mutate: func(c *v1.EngineClass) { c.Spec.Objects = v1.ObjectSettings{ Backend: v1.ObjectBackendS3, @@ -1230,18 +1250,228 @@ func TestEngineClassFloorFollowsTheObjects(t *testing.T) { } runUpdateCases(t, engineClass, []updateCase[*v1.EngineClass]{ { - name: "a 9Gi class moving its files from S3 into the stream's bucket is refused", + name: "a 10Gi class moving its files from S3 into the stream's bucket is refused", before: func(c *v1.EngineClass) { inS3(c) - c.Spec.Data.Storage.Size = new(resource.MustParse("9Gi")) + c.Spec.Data.Storage.Size = new(resource.MustParse("10Gi")) }, mutate: func(c *v1.EngineClass) { c.Spec.Objects = v1.ObjectSettings{Backend: v1.ObjectBackendNATS} }, want: msgStorageFloorNATS, }, { - name: "a 12Gi class moving its files from the stream's bucket to S3 is accepted", - before: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("12Gi")) }, + name: "a 14Gi class moving its files from the stream's bucket to S3 is accepted", + before: func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = new(resource.MustParse("14Gi")) }, mutate: inS3, }, }) } + +// TestEngineClassStoredBelowTheFloor is what the API server does with a class +// stored below the floor on its data volume, as one written before the floor +// last rose is (Crewlet's code host's log, the eighth state log a fresh node +// reserves, moved it from 12Gi to 14Gi with the files in the stream's bucket +// and from 9Gi to 10Gi in S3): it keeps the class as it is, and holds a +// change to it to the rules as they are now, sparing what the change leaves +// as it was (the API server's validation ratcheting). With the files in the +// stream's bucket the floor is a rule on the whole spec, so any change to the +// spec is refused until the size reaches the floor, and one to the metadata +// alone is not; in S3 it is a rule on the size alone, so only a change of the +// size that leaves it below the floor is refused. DataStorage.size's doc and +// the installing and troubleshooting guides, which tell a platform to raise +// such a class, say so from this test. +func TestEngineClassStoredBelowTheFloor(t *testing.T) { + s3Floor, natsFloor := installedVolumeFloors(t) + under := func(floor resource.Quantity, by string) *resource.Quantity { + q := floor.DeepCopy() + q.Sub(resource.MustParse(by)) + return &q + } + prefix := strings.ToLower(namespacePrefix(newNamespace(t))) + nats := engineClass("", prefix+"-stored-nats") + nats.Spec.Data.Storage.Size = under(natsFloor, "1Gi") + inS3 := engineClass("", prefix+"-stored-s3") + inS3.Spec.Objects = v1.ObjectSettings{ + Backend: v1.ObjectBackendS3, + S3: v1.S3ObjectSettings{Region: tenantsRegion, Bucket: tenantsBucket}, + } + inS3.Spec.Data.Storage.Size = under(s3Floor, "1Gi") + storeUnderLowerFloors(t, nats, inS3) + + window := func(c *v1.EngineClass) { c.Spec.Data.StartupSeconds = 3600 } + sized := func(size *resource.Quantity) func(*v1.EngineClass) { + return func(c *v1.EngineClass) { c.Spec.Data.Storage.Size = size } + } + for _, c := range []struct { + name string + class *v1.EngineClass + mutate func(*v1.EngineClass) + want string + }{ + { + name: "in the stream's bucket, a change to the spec is refused", class: nats, + mutate: window, want: msgStorageFloorNATS, + }, + { + name: "in the stream's bucket, a change to the labels alone is accepted", class: nats, + mutate: func(c *v1.EngineClass) { c.Labels = map[string]string{"example.com/tier": "team"} }, + }, + {name: "in the stream's bucket, the size raised to the floor is accepted", class: nats, mutate: sized(&natsFloor)}, + {name: "in S3, a change to the rest of the spec is accepted", class: inS3, mutate: window}, + { + name: "in S3, a size still below the floor is refused", class: inS3, + mutate: sized(under(s3Floor, "512Mi")), want: msgStorageFloor, + }, + {name: "in S3, the size raised to the floor is accepted", class: inS3, mutate: sized(&s3Floor)}, + } { + t.Run(c.name, func(t *testing.T) { + stored := &v1.EngineClass{} + if err := k8sClient.Get(ctx, client.ObjectKeyFromObject(c.class), stored); err != nil { + t.Fatal(err) + } + c.mutate(stored) + // A dry run, so every case meets the class as it was stored. + expect(t, k8sClient.Update(ctx, stored, client.DryRunAll), c.want) + }) + } +} + +// The EngineClass rules that floor the data volume, as controller-gen writes +// them into the CRD (internal/render's TestContractStreamFloor reads them the +// same way): the size's own rule, and the class's rule for a class whose +// objects are in the stream's bucket. +var ( + sizeFloorRule = regexp.MustCompile(`^!quantity\(string\(self\)\)\.isLessThan\(quantity\('(\d+Gi)'\)\)$`) + natsFloorRule = regexp.MustCompile(`^\(has\(self\.objects\) && has\(self\.objects\.backend\) && ` + + `self\.objects\.backend == 'S3'\) \|\| !has\(self\.data\.storage\.size\) \|\| ` + + `!quantity\(string\(self\.data\.storage\.size\)\)\.isLessThan\(quantity\('(\d+Gi)'\)\)$`) +) + +// engineClassCRD is the name of the EngineClass CRD. +const engineClassCRD = "engineclasses.crewlet.ai" + +// volumeFloorRules are the two rules of an EngineClass CRD's schema that +// floor the data volume: the size's own, and the class's for the files in the +// stream's bucket, each with the floor it names. It fails the test unless +// each is there exactly once, in the shape the patterns read. +func volumeFloorRules( + t *testing.T, schema *apiextensionsv1.JSONSchemaProps, +) (size, nats *apiextensionsv1.ValidationRule, s3Floor, natsFloor string) { + t.Helper() + spec := schema.Properties["spec"] + sizeProps := spec.Properties["data"].Properties["storage"].Properties["size"] + find := func( + rules apiextensionsv1.ValidationRules, pattern *regexp.Regexp, where string, + ) (*apiextensionsv1.ValidationRule, string) { + var found *apiextensionsv1.ValidationRule + var floor string + for i := range rules { + if m := pattern.FindStringSubmatch(rules[i].Rule); m != nil { + if found != nil { + t.Fatalf("%s: two rules of the shape %s at %s", engineClassCRD, pattern, where) + } + found, floor = &rules[i], m[1] + } + } + if found == nil { + t.Fatalf("%s: no rule of the shape %s at %s", engineClassCRD, pattern, where) + } + return found, floor + } + size, s3Floor = find(sizeProps.XValidations, sizeFloorRule, "spec.data.storage.size") + nats, natsFloor = find(spec.XValidations, natsFloorRule, "spec") + return size, nats, s3Floor, natsFloor +} + +// installedVolumeFloors are the floors on an EngineClass's data volume as the +// CRD envtest installed holds them: the size's own, which a class whose files +// are in S3 is held to, and the one for the files in the stream's bucket. +func installedVolumeFloors(t *testing.T) (s3, nats resource.Quantity) { + t.Helper() + for _, crd := range installedCRDs { + if crd.Name == engineClassCRD { + _, _, s3Floor, natsFloor := volumeFloorRules(t, crd.Spec.Versions[0].Schema.OpenAPIV3Schema) + return resource.MustParse(s3Floor), resource.MustParse(natsFloor) + } + } + t.Fatalf("no CRD %s was installed", engineClassCRD) + return resource.Quantity{}, resource.Quantity{} +} + +// storeUnderLowerFloors stores classes, each below a floor on the data +// volume, as an API server whose CRD had lower floors did: it installs the +// EngineClass CRD with both floors at 1Gi, creates the classes once the API +// server holds them to that, and installs the CRD's own rules again, +// returning once the API server holds them to those. No other test runs +// meanwhile, since none of this package runs in parallel with another. +func storeUnderLowerFloors(t *testing.T, classes ...*v1.EngineClass) { + t.Helper() + key := client.ObjectKey{Name: engineClassCRD} + installed := &apiextensionsv1.CustomResourceDefinition{} + if err := k8sClient.Get(ctx, key, installed); err != nil { + t.Fatal(err) + } + own := installed.Spec.DeepCopy() + lowered := installed.Spec.DeepCopy() + size, nats, s3Floor, natsFloor := volumeFloorRules(t, lowered.Versions[0].Schema.OpenAPIV3Schema) + size.Rule = strings.Replace(size.Rule, "quantity('"+s3Floor+"')", "quantity('1Gi')", 1) + nats.Rule = strings.Replace(nats.Rule, "quantity('"+natsFloor+"')", "quantity('1Gi')", 1) + install := func(spec *apiextensionsv1.CustomResourceDefinitionSpec) error { + return retry.RetryOnConflict(retry.DefaultRetry, func() error { + crd := &apiextensionsv1.CustomResourceDefinition{} + if err := k8sClient.Get(ctx, key, crd); err != nil { + return err + } + crd.Spec = *spec.DeepCopy() + return k8sClient.Update(ctx, crd) + }) + } + restored := false + t.Cleanup(func() { + if !restored { + if err := install(own); err != nil { + t.Errorf("installing the EngineClass CRD's own rules again: %v", err) + } + } + }) + if err := install(lowered); err != nil { + t.Fatalf("installing the EngineClass CRD with lower floors: %v", err) + } + // Asked of a name no class has, so that its create is judged by the + // rules alone, never refused as one that exists. + probe := classes[0].DeepCopy() + probe.Name += "-probe" + awaitAdmission(t, probe, true) + for _, c := range classes { + if err := k8sClient.Create(ctx, c); err != nil { + t.Fatalf("storing EngineClass %s under the lower floors: %v", c.Name, err) + } + } + if err := install(own); err != nil { + t.Fatalf("installing the EngineClass CRD's own rules again: %v", err) + } + restored = true + awaitAdmission(t, probe, false) +} + +// awaitAdmission waits, for up to half a minute, until the API server's +// verdict on creating class is admitted, asking with dry runs: a CRD whose +// rules changed is held to them once the API server has rebuilt its +// validation, not when the update returns. +func awaitAdmission(t *testing.T, class *v1.EngineClass, admitted bool) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for { + err := k8sClient.Create(ctx, class.DeepCopy(), client.DryRunAll) + if err != nil && !apierrors.IsInvalid(err) { + t.Fatalf("a dry run of EngineClass %s: %v", class.Name, err) + } + if (err == nil) == admitted { + return + } + if time.Now().After(deadline) { + t.Fatalf("the API server's verdict on EngineClass %s is still %v after the CRD changed, want admitted %v", + class.Name, err, admitted) + } + time.Sleep(250 * time.Millisecond) + } +} diff --git a/test/crd/fixtures_test.go b/test/crd/fixtures_test.go index 2d19328..23d26cc 100644 --- a/test/crd/fixtures_test.go +++ b/test/crd/fixtures_test.go @@ -150,6 +150,9 @@ const ( msgDashboardBaseURLNumber = "dashboardBaseURL's host ends in a number" msgDashboardBaseURLHost = "dashboardBaseURL's host must be a name, an IPv4 address or an IPv6 address" msgPublicBaseURLHost = "publicBaseURL's host must be a name, an IPv4 address or an IPv6 address" + msgPublicBaseURLPercent = "publicBaseURL's host may not hold a '%'" + msgPublicBaseURLName = "publicBaseURL's host may not hold a '<', a '>' or a ']'" + msgPublicBaseURLNumber = "publicBaseURL's host ends in a number" msgHostnameNumber = "hostname may not end in a label that is a number" msgS3RegionBlank = "region may not be white space alone" msgS3BucketBlank = "bucket may not be white space alone" @@ -164,9 +167,9 @@ const ( msgUnitID = "a Unit's name is its Crewlet unit id" msgDuplicate = "Duplicate value" msgThreeReplicas = "Unsupported value: 3" - msgStorageFloor = "size must be at least 9Gi" - msgStorageFloorNATS = "data.storage.size must be at least 12Gi with objects.backend NATS" - msgStartupFloorS3 = "data.startupSeconds must be at least 1776 with objects.backend S3" + msgStorageFloor = "size must be at least 10Gi" + msgStorageFloorNATS = "data.storage.size must be at least 14Gi with objects.backend NATS" + msgStartupFloorS3 = "data.startupSeconds must be at least 1601 with objects.backend S3" msgSetupFileTilde = "path may not start with ~ followed by a name, such as ~root" msgSetupFileBlank = "path may not be white space alone" msgSetupStepName = "name may not be white space alone" diff --git a/test/crd/main_test.go b/test/crd/main_test.go index 6ce9fd1..c25837c 100644 --- a/test/crd/main_test.go +++ b/test/crd/main_test.go @@ -101,6 +101,12 @@ func run(m *testing.M) int { fmt.Fprintln(os.Stderr, err) return 1 } + // For the one test that installs the EngineClass CRD with other rules + // for a while (storeUnderLowerFloors). + if err := apiextensionsv1.AddToScheme(scheme); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } if k8sClient, err = client.New(cfg, client.Options{Scheme: scheme}); err != nil { fmt.Fprintln(os.Stderr, err) return 1 diff --git a/test/crd/optionalkeys_engine_test.go b/test/crd/optionalkeys_engine_test.go index 9debb67..b008c73 100644 --- a/test/crd/optionalkeys_engine_test.go +++ b/test/crd/optionalkeys_engine_test.go @@ -53,7 +53,7 @@ func TestSetupStepOptionalKeysFollowTheEngine(t *testing.T) { steps[i] = v1.SandboxSetupStep{ Name: fmt.Sprintf("s%02d", i), Files: []v1.SandboxFile{{Path: fileNPMRC, ValueFrom: fromSecret("company-sandbox", "npmrc")}}, - Env: []v1.SecretEnvVar{{Name: envNPMToken, ValueFrom: fromSecret("company-sandbox", "token")}}, + Env: []v1.SandboxEnvVar{{Name: envNPMToken, ValueFrom: fromSecret("company-sandbox", "token")}}, OptionalKeys: keys, } } diff --git a/test/crd/optionalkeys_test.go b/test/crd/optionalkeys_test.go index 04df5c2..6001b34 100644 --- a/test/crd/optionalkeys_test.go +++ b/test/crd/optionalkeys_test.go @@ -135,7 +135,7 @@ func TestSetupStepOptionalKeysValidation(t *testing.T) { return v1.SandboxSetupStep{ Name: "registry", Files: []v1.SandboxFile{{Path: fileNPMRC, ValueFrom: fromSecret("company-sandbox", "npmrc")}}, - Env: []v1.SecretEnvVar{{Name: envNPMToken, ValueFrom: fromSecret("company-sandbox", "token")}}, + Env: []v1.SandboxEnvVar{{Name: envNPMToken, ValueFrom: fromSecret("company-sandbox", "token")}}, OptionalKeys: keys, } } diff --git a/test/crd/orgchart_test.go b/test/crd/orgchart_test.go index 77ab341..bf7d599 100644 --- a/test/crd/orgchart_test.go +++ b/test/crd/orgchart_test.go @@ -113,10 +113,10 @@ func fullAgent(s *v1.Seat) { s.Spec.Paused = new(true) s.Spec.PauseReason = "Deploy freeze until the incident review" s.Spec.StopRunning = new(true) + s.Spec.Code = v1.SeatCode{Access: v1.CodeAccessReview} s.Spec.Integrations = v1.SeatIntegrations{ Slack: v1.SlackSeatIntegration{CredentialsSecretRef: v1.SecretReference{Name: "ana-slack"}}, GitHub: v1.GitHubSeatIntegration{ - Tier: v1.GitHubAccessReview, Repos: []string{"nimbus-hq/nimbuscore"}, App: v1.GitHubApp{ ID: 1234567, @@ -190,6 +190,17 @@ func TestSeatValidation(t *testing.T) { want: msgAgentContact, }, + // Code access, in the API's own spelling. + { + name: "a code access tier on an agent seat is accepted", + mutate: func(s *v1.Seat) { s.Spec.Code = v1.SeatCode{Access: v1.CodeAccessFullAccess} }, + }, + { + name: "Crewlet's own spelling of a code access tier is refused", + mutate: func(s *v1.Seat) { s.Spec.Code = v1.SeatCode{Access: "full_access"} }, + want: `spec.code.access: Unsupported value: "full_access"`, + }, + // Contact accounts, each in its platform's own form. { name: "a human seat with every contact account is accepted", @@ -363,6 +374,13 @@ func TestSeatValidation(t *testing.T) { mutate: asHuman(func(s *v1.Seat) { s.Spec.Sandbox = &v1.SeatSandbox{} }), want: msgHumanForbidden, }, + { + // A person acts as their own identity on every code host, so a + // tier on their seat would cap nothing, and Crewlet refuses one. + name: "a code access tier on a human seat is refused", + mutate: asHuman(func(s *v1.Seat) { s.Spec.Code = v1.SeatCode{Access: v1.CodeAccessReadOnly} }), + want: msgHumanForbidden, + }, { name: "integrations on a human seat are refused", mutate: asHuman(func(s *v1.Seat) { diff --git a/test/crd/runenv_engine_test.go b/test/crd/runenv_engine_test.go new file mode 100644 index 0000000..b9326fb --- /dev/null +++ b/test/crd/runenv_engine_test.go @@ -0,0 +1,117 @@ +package crd_test + +import ( + "fmt" + "os" + "slices" + "strings" + "testing" + + v1 "github.com/crewlet/crewlet-operator/api/v1alpha1" + "github.com/crewlet/crewlet-operator/internal/engine" + "github.com/crewlet/crewlet-operator/internal/engine/enginetest" + "github.com/crewlet/crewlet-operator/internal/render" +) + +// reservedRunEnvFile is the engine's file whose ReservedSandboxEnv names the +// variables Crewlet sets in a coding run itself, which its config refuses in +// every env a run's environment is assembled from. +const reservedRunEnvFile = "internal/config/sandbox.go" + +// TestReservedRunVariablesFollowTheEngine holds the API's refusal of the +// variables Crewlet sets in a coding run itself to Crewlet's, both ways, in +// each env a run's environment is assembled from (runEnvPlaces): the API +// refuses a variable there exactly when crewlet validate refuses the +// document it renders at that variable. The names offered are the ones the +// engine's source reserves (its config.ReservedSandboxEnv), so a name the +// engine comes to reserve fails here until the API refuses it too; the ones +// the API's rules state (reservedRunEnv, which +// TestReservedRunVariablesValidation tries each rule with), so a name the +// engine reserves no longer fails until the rules let it be; and the ones +// beside them (ownRunEnv), so a rule that refuses a name the engine takes +// fails as well. The two lists are held to each other besides. +func TestReservedRunVariablesFollowTheEngine(t *testing.T) { + source, err := os.OpenRoot(enginetest.Source(t)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = source.Close() }) + src, err := source.ReadFile(reservedRunEnvFile) + if err != nil { + t.Fatal(err) + } + reserved := stringsOf(t, reservedRunEnvFile, src, "ReservedSandboxEnv") + if !slices.Equal(reserved, reservedRunEnv) { + t.Errorf("the engine reserves %v in a coding run's environment, and TestReservedRunVariablesValidation "+ + "tries the API's rules with %v", reserved, reservedRunEnv) + } + names := slices.Concat(reserved, reservedRunEnv, ownRunEnv) + slices.Sort(names) + names = slices.Compact(names) + + // Crewlet's verdict, on one document giving every name in each place: + // each refusal names the variable it refuses, at its own path. + in := sandboxCompany(setupStepWithEnv(names...), &v1.SeatSandbox{ + Env: seatSandboxEnv(names...), + Setup: setupStepWithEnv(names...), + }) + crewletRefuses := make([]map[string]bool, len(runEnvPlaces)) + for i := range crewletRefuses { + crewletRefuses[i] = map[string]bool{} + } + for _, p := range validateProblems(t, in) { + place, name, ok := runEnvPlaceOf(p.Segments) + if !ok { + t.Fatalf("crewlet validate refused the document elsewhere than this test expects (%s): %s", + render.PathOf(p.Segments), p.Message) + } + crewletRefuses[place][name] = true + } + + // The API's, on one object for each place and name. + namespace := newNamespace(t) + refused, offered := 0, 0 + for i, place := range runEnvPlaces { + for j, name := range names { + offered++ + apiAdmits := admitted(t, k8sClient, place.give(namespace, fmt.Sprintf("e%d-%02d", i, j), name)) + if !apiAdmits { + refused++ + } + switch crewlet := crewletRefuses[i][name]; { + case apiAdmits && crewlet: + t.Errorf("%s in %s: the API admits it, and crewlet validate refuses it, so the company is applied "+ + "nowhere", name, place.name) + case !apiAdmits && !crewlet: + t.Errorf("%s in %s: the API refuses it, and crewlet validate takes it", name, place.name) + } + } + } + if refused == 0 || refused == offered { + t.Errorf("the API refuses %d of %d variables: its rules are tried in one direction alone", refused, offered) + } +} + +// runEnvPlaceOf is which of runEnvPlaces a refusal of crewlet validate at +// path s is in, by its index, and the variable it names there: the place +// whose crewlet path is s up to its last step, a map key naming the +// variable. +func runEnvPlaceOf(s []engine.Segment) (place int, name string, ok bool) { + if len(s) == 0 || s[len(s)-1].IsIndex { + return 0, "", false + } + shape := make([]string, len(s)-1) + for i, step := range s[:len(s)-1] { + shape[i] = step.Key + if step.IsIndex { + shape[i] = "#" + } + } + at := strings.Join(shape, ".") + for i, p := range runEnvPlaces { + if p.crewlet == at { + return i, s[len(s)-1].Key, true + } + } + return 0, "", false +} diff --git a/test/crd/runenv_test.go b/test/crd/runenv_test.go new file mode 100644 index 0000000..569c1b9 --- /dev/null +++ b/test/crd/runenv_test.go @@ -0,0 +1,136 @@ +package crd_test + +import ( + "fmt" + "strings" + "testing" + + "sigs.k8s.io/controller-runtime/pkg/client" + + v1 "github.com/crewlet/crewlet-operator/api/v1alpha1" +) + +// msgReservedRunEnv is the message of the rules that refuse, in an env a +// coding run's environment is assembled from, a variable Crewlet sets in the +// run itself. +const msgReservedRunEnv = "are the engine's: Crewlet sets them in a coding run itself" + +var ( + // reservedRunEnv are the variables Crewlet sets in a coding run itself, + // its config.ReservedSandboxEnv, which + // TestReservedRunVariablesFollowTheEngine reads from the engine's source. + reservedRunEnv = []string{ + "CREWLET_GIT_TOKEN", "CREWLET_GIT_URL", "CREWLET_GIT_TRAILER_SEAT", "CREWLET_GIT_TRAILER_RUN", + "CREWLET_GIT_TRAILER_WORK_ITEM", + } + // ownRunEnv are names beside them that a run's environment may hold: a + // name Crewlet's own test of its rule takes, a prefix of the trailers' + // names and an extension of the token's, the variable that tells a node + // its boxes' git address (CREWLET_BOX_GIT_URL, which the engine reads from + // the node's environment, never a run's), the count git reads its + // configuration from the environment by, which Crewlet keeps and appends + // to, the token's name in another case, and a token for another git host. + ownRunEnv = []string{ + "CREWLET_GIT_PROXY", "CREWLET_GIT_TRAILER", "CREWLET_GIT_TOKEN_FILE", "CREWLET_BOX_GIT_URL", + "GIT_CONFIG_COUNT", "crewlet_git_token", "GITLAB_TOKEN", + } +) + +// runEnvPlace is one env a coding run's environment is assembled from: the +// field of a variable there that the API refuses one of reservedRunEnv at, +// the place the render writes the env to in the company document, where +// crewlet validate refuses one (its path up to the variable, a list index +// written #), and the object that gives one variable there. +type runEnvPlace struct { + name string + at string + crewlet string + give func(namespace, object, variable string) client.Object +} + +// runEnvPlaces are the envs a coding run's environment is assembled from: a +// Company's setup step's, a Seat's setup step's and the Seat's own, each of +// which Crewlet refuses a variable it sets in the run itself in. +var runEnvPlaces = []runEnvPlace{ + { + name: "a Company's setup step", at: "spec.sandbox.setup[0].env[0].name", + crewlet: "providers.sandbox.setup.#.env", + give: func(namespace, object, variable string) client.Object { + c := company(namespace, object) + c.Spec.Sandbox = v1.SandboxSettings{ + E2B: v1.E2BSandbox{APIKeySecretRef: secretKey("e2b", apiKeyKey)}, + Setup: setupStepWithEnv(variable), + } + return c + }, + }, + { + name: "a Seat's setup step", at: "spec.sandbox.setup[0].env[0].name", + crewlet: "roles.#.sandbox.setup.#.env", + give: func(namespace, object, variable string) client.Object { + s := seat(namespace, object) + s.Spec.Sandbox = &v1.SeatSandbox{Setup: setupStepWithEnv(variable)} + return s + }, + }, + { + name: "a Seat's sandbox env", at: "spec.sandbox.env[0].name", + crewlet: "roles.#.sandbox.env", + give: func(namespace, object, variable string) client.Object { + s := seat(namespace, object) + s.Spec.Sandbox = &v1.SeatSandbox{Env: seatSandboxEnv(variable)} + return s + }, + }, +} + +// setupStepWithEnv is one setup step whose env gives each of variables, read +// from a Secret. +func setupStepWithEnv(variables ...string) []v1.SandboxSetupStep { + env := make([]v1.SandboxEnvVar, len(variables)) + for i, name := range variables { + env[i] = v1.SandboxEnvVar{Name: name, ValueFrom: fromSecret("company-sandbox", "value")} + } + return []v1.SandboxSetupStep{{Name: "git", Env: env}} +} + +// seatSandboxEnv is a Seat's spec.sandbox.env giving each of variables, read +// from a Secret. +func seatSandboxEnv(variables ...string) []v1.SeatEnvVar { + env := make([]v1.SeatEnvVar, len(variables)) + for i, name := range variables { + env[i] = v1.SeatEnvVar{Name: name, ValueFrom: seatFromSecret("coder-sandbox", "value")} + } + return env +} + +// TestReservedRunVariablesValidation covers the API's refusal of the +// variables Crewlet sets in a coding run itself, in each env a run's +// environment is assembled from: each is refused, with the rule's message, at +// the variable's own name, where Crewlet refuses it too (at +// roles[i].sandbox.env., for one), and every name beside them is taken +// (TestReservedRunVariablesFollowTheEngine holds the rules to the engine). +func TestReservedRunVariablesValidation(t *testing.T) { + namespace := newNamespace(t) + objects := 0 + object := func() string { + objects++ + return fmt.Sprintf("r%02d", objects) + } + for _, place := range runEnvPlaces { + for _, variable := range reservedRunEnv { + t.Run(variable+" in "+place.name+" is refused", func(t *testing.T) { + err := k8sClient.Create(ctx, place.give(namespace, object(), variable)) + expect(t, err, msgReservedRunEnv) + if !strings.Contains(err.Error(), place.at+": ") { + t.Errorf("refused elsewhere than at %s: %v", place.at, err) + } + }) + } + for _, variable := range ownRunEnv { + t.Run(variable+" in "+place.name+" is accepted", func(t *testing.T) { + expect(t, k8sClient.Create(ctx, place.give(namespace, object(), variable)), "") + }) + } + } +} diff --git a/test/crd/sandbox_engine_test.go b/test/crd/sandbox_engine_test.go index cc7d93e..1dd2aff 100644 --- a/test/crd/sandbox_engine_test.go +++ b/test/crd/sandbox_engine_test.go @@ -125,7 +125,7 @@ type stepShape struct { // and a step that does one thing of each kind Crewlet counts. func setupStepShapes() []stepShape { file := []v1.SandboxFile{{Path: ".npmrc", ValueFrom: fromSecret("company-sandbox", "file")}} - env := []v1.SecretEnvVar{{Name: "NPM_TOKEN", ValueFrom: fromSecret("company-sandbox", "token")}} + env := []v1.SandboxEnvVar{{Name: "NPM_TOKEN", ValueFrom: fromSecret("company-sandbox", "token")}} return []stepShape{ {"a name of a space", v1.SandboxSetupStep{Name: " ", Brief: "b"}}, {"a name of tabs and a wide space", v1.SandboxSetupStep{Name: "\t\u3000\t", Brief: "b"}}, diff --git a/test/e2e/company_test.go b/test/e2e/company_test.go index ea00d8f..b27b6a7 100644 --- a/test/e2e/company_test.go +++ b/test/e2e/company_test.go @@ -102,7 +102,7 @@ var _ = Describe("Company", Ordered, func() { // The smallest volume the API admits. kind's // provisioner does not enforce it; the engine's // event stream is sized from it all the same. - Storage: v1.DataStorage{Size: new(resource.MustParse("12Gi"))}, + Storage: v1.DataStorage{Size: new(resource.MustParse("14Gi"))}, }, Exposure: v1.ExposureSettings{Webhooks: v1.WebhookExposure{Mode: v1.WebhookExposureNone}}, NetworkPolicy: v1.NetworkPolicySettings{ diff --git a/test/e2e/pools_test.go b/test/e2e/pools_test.go index aa1a32d..d903d11 100644 --- a/test/e2e/pools_test.go +++ b/test/e2e/pools_test.go @@ -120,7 +120,7 @@ var _ = Describe("Seat pools", Ordered, func() { Engine: v1.EngineSettings{Image: crewletImage, ImagePullPolicy: corev1.PullNever}, Data: v1.DataPodSettings{ Resources: small, - Storage: v1.DataStorage{Size: new(resource.MustParse("12Gi"))}, + Storage: v1.DataStorage{Size: new(resource.MustParse("14Gi"))}, }, SeatPools: v1.SeatPoolSettings{ Policy: v1.SeatPoolsAllowed, @@ -228,8 +228,10 @@ var _ = Describe("Seat pools", Ordered, func() { expectSeatsFollowTheFleet() }) - // The data pod's leaf listener takes any connection, with no - // authentication: -broker is what keeps it the company's. + // -broker is what lets a pod reach the data pod's leaf listener + // at all, beside the key from the company's keyring a link must present. + // The probe reads the INFO line the listener sends before any + // authentication, so it shows the port reachable, not a link admitted. It("admits the company's own seat pods to the data pod's leaf listener", func() { Expect(leafProbe(poolsNamespace, "leaf-own", seatLabels(poolsCompany), leafAddress)).To(Equal(corev1.PodSucceeded), "a pod labelled as the company's seat pod did not join the leaf listener: the probe proves nothing")