Repository navigation
feat(api): follow Crewlet's Chatlet and its nine state logs - #52
Merged
Merged
Conversation
3 tasks done
The engine writes what a person reads as sentences, each ending with a
full stop: a refusal's message, detail and hint, and its 503 sentences.
An error of Go's or of the operator's own has none. Some conditions and
events end by quoting one of those, and each site decided on its own
whether to add a full stop, so they disagreed:
- Sites that added one ended in two whenever the quote was the engine's.
These are the EngineUnavailable condition ("...or ask another
node.."), SecretsSynced after a refused store write, a pause that may
not have landed, ApplyFailed's account and its unreadable fleet
record, and the Reactivated warning of a reload the operator could
not read back. The last quotes mid-sentence, so it ended in "..
It takes".
- Sites that added none ended with no stop whenever the quote had none:
Go's errors at EngineHealthy, IdentityClaimed and a Seat's Ready when
the health or the fleet could not be read, and at a pause refusal the
engine's account of the refused call when it ended without one.
- A Paused message quotes the reason the person who paused the seat
gave. That reason is their own words, so "Hold the deploy." became
"Hold the deploy.. It starts no new turn".
There is now one rule, ended: add a full stop where the text has none,
and leave text that already ends with ".", "!" or "?", or with a
parenthesis that closes on one, as it is. The first half is the rule the
engine ends its own details and hints by (httpjson.Sentence). The second
is for the client's quote of a refusal, which puts the engine's hint in
parentheses after its message: the hint is a sentence with its own full
stop, so a message quoting one already ends inside them, and a stop
after them made it "....).". sentences now uses the rule too. Every site
above ends through it, and so does each message that ended with a list
or a render finding plus a "." of its own, so the package has one way to
end a sentence.
TestQuotedEndingsGoThroughEnded walks the package and refuses a message
that ends with `+ "."` or with a format ending ": %s." or ": %v.", so a
new site can't add the old shapes back. Its comment says what it can't
see: a quote in the middle of a message, which the tests beside it hold
one by one.
TestEndedEndsASentenceOnce covers the rule, a closing parenthesis
included. These cover the sites, each with a quote that ends in a full
stop and one that doesn't, and the conditions' test with a quote that
ends in the engine's hint too (the 503 a starting node answers):
- TestAConditionThatQuotesAnErrorEndsItOnce
- TestAFailedApplyAccountEndsWhatItQuotesOnce
- TestASyncThatFailsEndsWhatItQuotesOnce
- TestAReloadThatCannotBeReadBackEndsWhatItQuotesOnce
- TestAPauseVerdictEndsWhatItQuotesOnce
- TestAPauseReasonIsQuotedAsWrittenOnce
Each site test and the walk fail against the sites as they were, with
ended defined beside them, and pass with this change; the conditions'
hint case and the rule's own fail with ended blind to the parenthesis.
The fake engines gained what those tests need: a /fleet status, a store
and a document read that refuse as an engine that can't decide who is
asking does (503 identity_unavailable, its request guard's answer), and
pause tools that answer an unknown outcome. Their refusal now carries
the engine's words, not a sentence the engine never wrote.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The pause tests' engine answered a roster that cannot be read with
http.Error: a text/plain body holding {"error":"unavailable"} and no
Retry-After. The engine never gives that answer. Once a node serves,
GET /agents fails only where it cannot decide who is asking (its
auth.Caller), and then answers 503 identity_unavailable as JSON, with
its sentence and Retry-After: 2, on crewlet's main and at crewlet#142
alike. The client read the double's answer as a proxy's page rather than
as the engine's refusal, so a change to how the operator treats the
engine's answer would have been tested against a page instead.
The double now answers as the engine words it.
TestAPauseThatCannotBeMadeIsAskedForAgainSoon holds what it held: the
pause is taken without the roster, and a look comes again within ten
seconds, which is what the operator does with any roster it cannot read.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
TestPutSecretErrorsNeverContainTheValue refused the write as 503 no_keyring with a detail and no message, and failed it as 500 internal_error with no message. The engine gives neither. Every node holds the keyring now: its secret surface refuses to start without one, so no route of it answers no_keyring, on crewlet's main and at crewlet#142 alike. And every refusal carries the sentence the engine writes for its code. A test of what an error may say was checking an error the client never builds. The refusal is now the 413 body_too_large the route answers a value past what it takes, and the 500 carries the engine's sentence, as the engine words both. The test holds what it held: no error carries the value. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The engine answers a read of the fleet's company document 409
unreadable_revision when that revision is sealed under a key its keyring
no longer holds, and 409 unsealed_revision when the revision is stored
without a seal, which no node reads. It answers a write the same way,
since a write reads the revision it replaces first. Each carries the
revision, the engine's account as its detail, its remedy as its hint,
and no Retry-After. The code's own sentence says to put the key back in
the node's keyring (crewlet's main and crewlet#142 alike).
The operator read that as an engine it could not reach: ConfigApplied
False EngineUnavailable, "the operator's request to the company's engine
failed, and it tries again in 10s", and a look every ten seconds. No
look clears it, and nothing the operator sends does. What does is given
to the data pods: a key put back in Secret <company>-keyring, which the
operator renders into every pod and never rewrites, and which the pods
read only when they start.
So the client tells it apart (ErrRevisionUnreadable, matched by the two
codes, with the revision decoded into APIError.RevisionID), and a
reconcile it ends reports ConfigApplied False RevisionUnreadable, with
one Warning event: the revision, the engine's refusal quoted whole, and
what restores it. For unreadable_revision that is the key put back in
the keyring Secret from the backup kept with the volumes. For
unsealed_revision, which no key opens, it is the engine's hint. The next
look is at the resync interval, as for an engine with nothing to do,
rather than ten seconds on. The data pods restarting with a restored key
changes RuntimeReady, which runs the reconcile at once. The engine's
failures that end a reconcile are now decided in one place
(engineFailed), and their next look in one (engineRetry).
The withdrawal of a deleted Secret's value met the same answer and said
it would try again in ten seconds. It now waits as engineRetry says, and
follows a Retry-After the engine sends, which it used to ignore. Its
SecretWithdrawalFailed event now ends through ended(), like the
conditions that quote the engine.
The troubleshooting guide gains the reason's row, its branch of the
flowchart, and "The engine cannot open its company document";
applying.md lists the reason.
TestARevisionTheEngineCannotOpenBlocksTheDocument fails against the old
reconcile: EngineUnavailable, a promise of a retry, and a look ten
seconds on. Its doubles send the detail and the hint as the engine's
httpjson.Sentence ends them, and it holds the message to quoting the
hint, which closes on its own full stop, without a second one after it
("....)."). It also fails with the reason's branch dropped, with the
next look back on retryAfter, and with one remedy for both codes.
TestAWithdrawalOverARevisionTheEngineCannotOpenWaitsForIt fails with the
withdrawal back on engineRetryInterval.
TestAnUnreadableRevisionNamesItAndItsRemedy fails with the revision left
undecoded, and TestErrorMapping holds both codes to the sentinel and a
proxy's 409 to none.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet's main has regrouped its dashboard, and crewlet#142 keeps it so: the org chart and the schedules moved under Company (Company › Org chart, Company › Schedules), and Settings › People & access split into Settings › People, Settings › Service accounts and Settings › API tokens. The getting-started guide and the people guide still sent somebody to Agents › Org chart to claim the founder seat and invite the others, and to Settings › People & access to change them, and the company document page to Agents › Schedules for a schedule's history: screens the dashboard no longer has. They now name Company › Org chart, Settings › People and Company › Schedules, as Crewlet's own guides do. The labels on the way there (Sign in with token, Use an API token instead, Open seat, Invite, Add person) are as they were. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The runtime page's backup command ran crewlet backup in the data pod with no token. Every command that reaches a running node takes its token from CREWLET_API_TOKEN and nothing else, never a flag, and on the main image, which resolves no request without a credential (only an unstamped development build run with -dev-principal does, and only on loopback), the node answered the backup 401, so none was taken (crewlet's main and crewlet#142 alike). The command now pipes the access token the Company publishes into CREWLET_API_TOKEN inside the pod, as the people guide's invitation does, and passes the backup's directory as an argument expanded on the operator's machine, so the timestamp's quoting survives the single-quoted shell. The node is named with -url, the flag crewlet's main gives it. Checked: crewlet's main binary takes -url and refuses -api, and under dash, which the engine image's Debian base runs as sh, the token reaches crewlet's environment and the directory its -dir. The full command was not run against a cluster. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Two of the engine client's doubles answered with words crewlet's main never writes: - The 503 every route but the probes answers while a node starts said "This node is starting." with a hint of its own. The engine says "This node is still starting and is not serving yet. Try again in a moment.", with what it is recovering as the detail and the hint that names /health's status and /ready's reason, each capitalised and ended with a full stop, as httpjson.Sentence ends every refusal's detail and hint. - An unreadable lease table on GET /fleet carried "the lease table could not be read" as its message and no Retry-After. The engine answers a coordination store that did not answer with the unavailable code's own sentence and a Retry-After of five seconds, the health tick's. No operator code reads those words, so what each test decides is as it was: ErrUnavailable with the engine's Retry-After and not ErrNotReady, and an error rather than an empty fleet. They now describe an engine somebody runs. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The double that stands in for the engine in the secret withdrawal tests said two things the engine never says. Its /health posture was "serving", where the engine's postures are serve, wait, shed, isolated and stuck, and it refused an invalid document with 422, where the engine writes 400 for validation_error. Neither changed what the tests decide, since the client reads the posture through Diverged and a refusal by its code, but a double that disagrees with the engine certifies a client against an engine nobody runs. Answer the posture as engine.PostureServe and the refusal as a 400, as crewlet's main and crewlet#142 alike answer them. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
A Company deleted under spec.deletionPolicy Retain keeps its data volume claim, and a Company created again under its name runs on that claim as it is: its StatefulSet names the same claim, so the pod binds it whatever the new template asks for. dataVolumeSize counted every claim at no less than the template it took for the claim's maker, so where the class had been raised in between (a 12Gi claim kept, the class now 16Gi), the engine's stream was sized for the 16Gi the template asks for, on the 12Gi volume the pod has, which the stream could outgrow. A StatefulSet makes each claim at its template's size, and a claim's request only ever grows, so one asking for less than the template is not the template's: it now counts at what it asks for, or at what it was given if less. Every claim the template made is counted as before. TestDataVolumeSize gains a kept claim under the template, before there is a StatefulSet, rounded up, given less than it asks, not bound yet and expanded past the template; the first five fail with the rule removed. The runtime page says how a kept claim is counted, and that a Company created again over one has its stream sized from it. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
When the activation a withdrawn value owes fails, SecretWithdrawalFailed says when the operator tries again: engineRetry of the error, which is the engine's own Retry-After (two seconds for an engine that cannot say who is asking), ten seconds for anything without one, and the resync interval for a revision the engine cannot open. The reconcile came back at another time. After a document the engine refused, settleRefused requeued ten seconds on whatever the event said, so a revision the engine cannot open, which no wait clears, was asked about again every ten seconds under an event saying five minutes; and a reconcile with nothing to write came back at the smaller of ten seconds and the resync interval. settle now returns the retry withdrawalFailed reported, zero where nothing failed, and both callers requeue at it: settleRefused as it is, and the reconcile with nothing to write no later than the resync interval; both still look within the engine's pointer poll while a value is kept for a revision other than the fleet's (heldResync). The reconcile after a write keeps its own two-second poll, which comes sooner. The storeEngine double can refuse a reload (reloadDown) as an engine that cannot say who is asking refuses one, with a Retry-After of two seconds. TestARefusedDocumentComesBackWhenItsWithdrawalReportSays holds a refused document whose activation then failed to the two seconds the event promises; it went red with settleRefused's ten seconds put back. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
When the engine runs the rendered document as a revision the operator
did not activate itself (a reload after the engine minted a credential,
or a person's crewlet config reload), the operator adopts it into
status.config without a write. It moved revisionID onto the new
revision and kept epoch as it was: the epoch of the revision it
replaced. verifyApplied takes that epoch for the one to wait for, so a
Company read the reload as applied as soon as the engine had applied
the revision before it, and its Applied event paired the new revision
with the old one's epoch ("applied revision r3 (epoch 2)"). The field
says it is the activation counter of the revision it names.
GET /config names the revision and no epoch, and /health the epoch a
node has applied, which may be another revision's; the fleet's
activation pointer, which GET /fleet answers beside its nodes, names
both. So the engine client reads it (Fleet.Activation), and the in-sync
path records the epoch it names for the adopted revision, from the
fleet the reconcile's secret sync has just read (keepServing). The
in-sync branch of enforce moves into a helper of its own (keepInSync),
which keeps enforce under the cyclomatic bound with the error it now
handles.
Waiting for that epoch makes the wait's deadline matter: verifyApplied
reports ApplyFailed once the engine has not applied the epoch 72 seconds
after status.config.appliedAt, and the adoption kept appliedAt as it
was, the operator's last write, another revision's and possibly days
old. A reload its node had not applied yet read as an engine that does
not poll its activation pointer the moment the operator saw it. So an
adoption records appliedAt as when the operator found it, the wait
counts from there, the field's doc says so, and ApplyFailed says the
operator activated the revision or found it activated.
A fleet that cannot be read records nothing and fails the reconcile,
which reads both again. So does one that answers without its activation
(the engine could not read its pointer for the answer), with an error
that claims no race, since nothing said another revision activated. GET
/config and the fleet's activation both read the fleet's pointer, so an
activation naming another revision is a write that activated one between
the two reads: a lost race (activationAhead, an ErrRevisionAdvanced),
which converge reads again from the start once, as it reads a write that
lost. The errors say the operator has not recorded the revision rather
than that it did not activate it, which it cannot know.
TestARevisionAdoptedInSyncIsRecordedAtItsEpoch fails with the epoch left
as it was, with appliedAt left as the last write's, with the
activation's revision unchecked, and with a fleet that cannot be read, a
missing activation or one naming another revision read as the other;
the envtest reload case holds status.config to the reload's own revision
and epoch, and the engine suite holds Fleet.Activation to the fleet's
revision. TestAnAdoptedRevisionIsHeldToItsOwnApplyDeadline fails with
appliedAt left as the last write's and with ApplyFailed saying only that
the operator activated the revision, and
TestAnAttemptReadsTheFleetAfterItsOwnReadOfTheDocument with the second
attempt judged against the fleet the first one read. The storeEngine
double gains hooks before a read of the document or of the fleet, the
activation pointer in its fleet's answer, an activation-less fleet and a
node behind on its applies. The applying concept page and the
troubleshooting guide say which epoch is recorded and when its wait
starts; and the guide's section on ApplyFailed now says it covers a
revision the operator adopted as well as one it wrote, which it already
did before this commit through the engine's own verdict of a diverged
apply.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
ReloadConfig's doc says a reload is a compare-and-set against the fleet's activation pointer, so a write landing inside the request fails it with ErrRevisionAdvanced, as any write that lost a race does (crewlet answers 409 revision_advanced, naming the revision it stored and the one that won). Nothing pinned that mapping for a reload: a reactivation that fails leaves its withdrawal owed and is reported, and the next reconcile decides again by the revision then active, so a reload read as another failure would go unnoticed. TestAReloadThatLosesItsRaceIsRevisionAdvanced pins it. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The engine reads its secret store only when a revision activates. When the sync writes back a value somebody overwrote, or one the store lost, the fingerprints stay as they were, and so does the hash of what the operator applied, but a revision activated since the overwrite may run the other value. The in-sync check caught that only through the sync of the same reconcile: it required that this reconcile's sync wrote nothing. A reconcile that wrote a value back and then did not get its write of the document through (a dry run the engine could not answer, a write that failed, a write that lost its race and the retry whose sync then had nothing left to write) left nothing behind that said so. The next look found the document, its values and the hash all matching, and took the engine for in sync while it went on running a revision that read the store before the write-back, until the next Secret change. So the state records it, as it records a withdrawal: every name Sync writes a value under stays in State.Written, saved in the state Secret under written-names, until a revision activates after it. And what settles either is no longer only an activation the operator makes and sees answered, which a save lost after it, or a restart, left owed: the next look wrote the same document again over a write that ran (an epoch every node applied for nothing), or reloaded the revision again for a withdrawal it had read. Every change is marked with the epoch of the fleet's activation pointer read first after it (State.Observed): any revision the pointer names at a greater epoch activated after the change, and every node reads the store at its attempt to apply it, a node that refuses the revision included, whose running revision resolves its references from what that attempt read. The operator reads the pointer, and saves the marks, before it asks for the write or the reload that is to read them (markChanges), and a change is settled once the pointer is read naming a revision above its mark (State.Ran, settleRead), whoever activated it: the operator's own write or reload, whose saved record was lost, a person's reload, or the engine's own reload after it mints a credential. settleRead takes the fleet that keepServing reads afresh after the reconcile's read of the document, and counts only the revision that read returned, as one it saw opened: GET /config and the activation both read the pointer, and a revision a write activated between the two reads may be one no node can open. An activation the operator makes and sees answered settles every change still (State.Activated, State.Reloaded), and a pointer it cannot read leaves the changes for one. The marks the read lowers are saved before any activation is asked for too: the sync saves them in a reconcile that applies, and a reconcile whose document is held saves them itself (State.SameMarks says the read moved one), since a save lost after its reload left a change loaded unread again, which no epoch is above. A value Sync or Withdraw deletes owes nothing, and nor does one the store no longer holds that nothing writes again. And a value is marked before it is written. Sync hands its caller the State with every name it is about to write recorded as written, and the fingerprints as they were, and writes nothing unless the caller saved it: recorded only after the write, a save lost between the two left a value no revision read and nothing that said so, and the next sync, finding the store holding the operator's own value under the fingerprint it had, had nothing left to write. A name saved as written whose value never reached the store costs an activation for nothing; saved under its new fingerprint, a rotated value that never reached the store would never be written at all. The engine re-activates the revision itself at a sync's last write or removal of a value it reads (secretsync.Result.Reload). That activation lands inside the request, after every change the sync made, so no read of the pointer can mark them first: it settles them from its answer once read back, as the operator's apply where the sync wrote a value (reloadedBySync, State.Activated) and as the withdrawal it owes where it only removed one (reactivate). A write whose re-activation failed leaves its values owed, for the operator's apply. With that the in-sync check no longer compares the hash it recorded: a revision of the rendered document the engine runs above every mark has read every value, whatever the operator recorded, a write of it whose record the reconcile that made it never saved included, and it is recorded with the hash of what it runs. And a reload that fails, or loses its race to another writer, is read back (reloadFailed): a revision above the withdrawal's mark settles it, recorded as the reactivation where it is the running document again (the request landed although its answer did not, or somebody reloaded it), and reported with a Warning Reactivated that the operator activated nothing again where it is another document. It was taken for a failed withdrawal: a Warning that the engine may still start a tool server with a value that no longer exists, and an epoch every node applied for nothing at the next look. The write's summary names the Secrets owed rather than only what this reconcile wrote, since an earlier one may have written them; and a reactivation the operator records is dated by its clock, as every other record is, not the process's. Sync's write-ahead goes into helpers of its own (toWrite, writeAhead), which keep it under the cyclomatic bound; State, with its new map, is 72 bytes, under the 80 gocritic holds a value parameter to. storeEngine can fail a dry run as an engine that cannot decide one, and runs a hook once a value is written into its store. TestAValueWrittenBackStaysOwedThroughAFailedWrite fails with the in-sync check back on this reconcile's writes (the reload is adopted and nothing written) and with the summary built from them. TestAWrittenValueIsOwedUntilARevisionActivatesAfterIt fails with Sync not recording a write, with Activated or Reloaded not settling one, with Reactivated settling one, with a deleted value kept owed, with one the store lost kept owed, and with the record not carried from one sync to the next. TestAChangeIsSettledByAnActivationAboveTheEpochReadAfterIt fails with a later read raising a mark, with an activation at the epoch read settling it, with a name written again keeping its mark, and with Unobserved blind to a change. TestSyncSavesAWriteBeforeItWritesTheValue fails with the save skipped and with the new fingerprint saved ahead of the write. TestSaveStateKeepsTheWrittenNames fails with the names or their epochs not saved or not loaded, and with a negative epoch read as one. In the controller, TestASaveLostAfterAConfirmedWriteWritesNothingAgain fails with the read settling nothing; TestAWriteBackWhoseSaveIsLostStaysOwed with the save before the write skipped; TestARevisionThatReadEveryValueIsInSyncWhateverHashWasRecorded with the hash compared again or not recorded; TestAFailedReloadIsReadBack with the read back settling nothing, with the running document again taken for another, and with the reactivation dated by the process's clock; TestAnActivationAfterAWithdrawalSettlesIt with the read settling nothing; TestAnActivationNoNodeCanOpenSettlesNothing with an activation other than the revision GET /config returned settling; TestAReloadIsRecordedOnlyForTheDocumentItRead with the unconfirmed reload settling withdrawals alone; and TestAHeldReconcileSavesTheMarksItsReadSetBeforeItReloads, whose fleet answers the read that says what every pod runs and not the one after the withdrawal, with the held reconcile's save of what its read marked skipped. The envtest of a deleted value let go while the engine runs a reload of the rendered document expects that one activation to be a reload of the running revision (Reactivated) now: with the recorded hash no part of the question, the engine is in sync, and the withdrawal is all it owes, where the operator used to write the same document again; so does TestTheNextDocumentLetsGoWhatTheRunningRevisionRequired, where the engine activates the running revision again at the removal. The tests that pin the operator's own activation after a write-back or a withdrawal make the engine's own re-activation fail, and TestAFailedReloadIsReadBack owes the values pushed after r1 explicitly (owePushed). newWithdrawal records r1 as activated after both values were pushed, as its comment says: it saved them as written and unread, which every reconcile would have written the document again for. The applying concept page says what the engine has read, and when a change stays a reason to write or to activate again, what a change no read of the pointer followed waits for, and what the state Secret holds; status.config.appliedHash's doc says what it records now. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The secret sync runs before the dry run, so by the time the engine refuses the rendered document the sync may have removed from the store the value of a deleted Secret that the revision the engine runs lets go without. The engine reads its store only when a revision activates, so the operator activates the running revision again for it (settle). It did that only for a document the engine found invalid. A document the engine refused with 409 seat_held (a human seat somebody holds that the document takes away), 403 config_managed (a token the engine does not let change the document) or a 403 the token lacks a grant for went back as an error, which engineFailed reported and nothing settled. For as long as the seat was held the engine went on starting tool servers and executor runs with the value of a Secret that no longer exists, while the applying concept page said that whenever the reconcile ends without an activation after it deleted a value, the operator activates the running revision again. Now every refusal the Company's conditions report as what holds the document is one (refusedByEngine), for the dry run and the write alike: the refusal is recorded as before (invalid, seatHeld, writeRefused, notGranted) and the activation owed is made (settleRefused). Neither a held seat nor a managed document refuses a reload of the running revision, which is admin's; a token without the grant meets the same refusal on the reload, which is reported as a failed withdrawal. A refusal the engine's own users clear is looked at again ten seconds on as before, since nothing in the cluster changes when they do, or sooner where a failed activation's report promises it; a document the engine finds invalid still waits for an object to change, or for the pointer poll while a value is kept for a revision other than the fleet's. storeEngine can refuse every write of the document with seat_held or config_managed (refusal). TestADocumentTheEngineHoldsStillSettlesTheStore fails with those refusals left to engineFailed, with the look ten seconds on lost or made later than a failed activation's report says, and with the activation not made. The applying concept page names the engine's refusals among what holds a document, and the troubleshooting guide's SeatHeld entry says what happens to a value removed while it holds. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The engine reads its secret store only when a revision activates. A value
the sync writes, a credential somebody rotated or a value written back
over one somebody else replaced, reaches the revision the engine runs
only through the write of the document that follows it. When the engine
refused that write (a document it finds invalid, a human seat somebody
holds, a token it does not let change the document), or the document was
held for a Secret, nothing followed: the operator activated the running
revision again for a withdrawal it owed and for nothing else. So a key
rotated because it leaked stayed in use by the revision the engine ran
for as long as the document was refused, while every node that
restarted read the new one.
Now reactivate owes the running revision an activation for every store
change it reads once it activates again (owedBy): a withdrawn name it
lets go without, as before, and a name written that it references (its
document's ${...} references). settle and a reconcile whose document is
held both make it. A written name the running revision does not
reference stays owed for the write that does. The summary, the
Reactivated events and the error of a failed request say which changes
the activation reads (storeChanges), and a failed activation owed only
for written values is reported as SecretRefreshFailed rather than as a
withdrawal that failed; one owed for a withdrawal is reported as before.
The engine activates the revision it runs again itself at a sync's write
of a value that revision reads (secretsync.Result.Reload), so this is
the activation where that one failed: reactivate takes the engine's
where it landed, as it does at a removal, and asks for one otherwise. The
tests that pin the operator's activation make the engine's fail.
TestAFailedReloadIsReadBack no longer owes the values pushed before r1
(owePushed, now gone): the reload it pins reads them too, and it expects
the withdrawal alone owed.
TestARotatedValueReachesTheRevisionTheEngineRunsWhileTheDocumentIsHeld
fails with written values left out of the activation owed, with settle
returning before it looks at them, with a value the running revision
does not reference activated for, and with a failed activation for one
reported as a failed withdrawal; TestAHeldDocumentHasTheEngineReadAValueWrittenBeforeIt
with the held reconcile returning before it looks at them, and with its
failed activation reported as a failed withdrawal. The applying concept
page says that a written value no write can follow is activated for,
and names SecretRefreshFailed; the credentials guide says a rotation
reaches the engine while it refuses the document.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The controller package's tests that drive a reconcile against an engine double spend most of their time building their fixtures rather than running the code they test: each parsed every file in config/samples again (decodeSamples, through optionalRun, called twice per withdrawal), built a runtime scheme of client-go's types and the crewlet.ai types, and gave the fake client no REST mapper, so it derived a static one from that scheme each time. Profiled, those three were half of the CPU of the package's tests outside the envtest suite. They run one after another beside the envtest suite in one test binary, which runs on go test's default ten-minute timeout, and on a 4-CPU machine the package already takes eight of those minutes. The samples are parsed once per test binary (samples, a sync.OnceValues) and each call of decodeSamples deep-copies them into its namespace, so no test shares an object with another; the scheme and its REST mapper are built once (testScheme, testRESTMapper), being read-only once built, and every fake client in the package takes them through fakeClient. Those tests took 23.3 s and take 12.8 s, measured with the same load on the machine. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
make test ran every package on go test's default timeout of ten minutes per test binary. internal/controller is one binary holding the envtest suite, which runs every config spec against a real engine, and the package took 478 s on a 4-CPU machine under the race detector, 80 per cent of that window. With more engine-double tests in the package it took 596 s, and a run beside another job crossed the window and failed with "test timed out after 10m0s" while every test was passing; a slower runner fails the same way. The e2e target already names its own (60m). The test target now passes -timeout 20m: twice the suite's measured time, so a runner half as fast still finishes, while a hung suite is still stopped well inside a CI job's limit. The comment beside the target says what the value is sized to. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
When the engine stores a written revision and then loses the answer to the compare-and-set that activates it, it answers PUT /config with 503 unavailable, "outcome": "unknown", the stored_revision_id and the node that stored it, under the Retry-After of an undecided answer (crewlet's RefuseActivationUnknown, on main and in crewlet#142 alike): the fleet may run the revision or may not. The operator returned that like any other failure and kept nothing of it. Where the write had landed, the next reconcile read a revision it had not recorded: where its document is the rendered one, it adopted it as a revision it did not write, and where the render had moved on since, it reported its own write as drift it corrected and read every value the write's document lets go as required. Where the write had not landed yet, the next reconcile sent it again, and if the first landed before the second was decided, the second lost the race to it. Under Seed the write that seeded the engine was reported as a document the engine had before the operator seeded one. Only a read can settle it, and two reads do: GET /config (the fleet's activation pointer, crewlet ADR-0057) names the fleet's revision, and GET /fleet's activation the epoch it runs it as. So the write is recorded in status.config.unconfirmed (its revision, the revision it was built on, its appliedHash, when, its object counts, and what it replaced where that was a revision somebody else wrote with other content: replaced Drifted, and driftedAt), and settled by the next reconcile before anything is decided (convergeOnce). The engine runs it: it is recorded as the operator's write at the activation's epoch, with an Activated event, and reported as a confirmed write is, a drift it corrected included (reportReplaced, which recordApply calls too); it is dated when the operator found it, since nothing says when its activation landed, so the engine is held to applying it from then, as an adopted revision is. A fleet that cannot be read, or answers without its activation, leaves it recorded and writes nothing; one whose activation names another revision is a lost race (activationAhead), which converge reads again from the start, as it does for an adopted revision; for that, each attempt reads the fleet after its own read of the document, since the confirmation runs before keepServing reads it afresh. The next look is when the engine's Retry-After says (engineRetry), ten seconds where it sends none. The engine runs the revision the write was built on: it stays, and the write is sent again on the same condition, which the engine refuses if the first one landed meanwhile. Any other: it can no longer land, and goes. A write into an engine that had no document moves to status.config.overtaken instead, for Seed, which takes it for the seed whatever revision runs now, since nothing the engine says tells a seed that landed and was replaced from another writer's create that won; it stays until a reconcile records a revision in sync or the seed, or confirms a write of the operator's running, beside which no create can land or be the seed. A write built on the one confirmed, made in the moment that one landed, stays unconfirmed. Both lists are bounded at the CRD's 16, the newest kept. What the write's activation read of the store is settled two ways. The store changes it read were marked with the fleet's epoch before it was sent, and a revision the engine runs above that epoch settles them, whoever activated it (settleRead). So a write that landed and was activated again before the next read (the engine's reload after it mints a credential, a person's crewlet config reload, the operator's own) is in sync where its document is the rendered one, and nothing activates it again. A change no read of the pointer followed, because the fleet could not be read, which is when the engine is likeliest to lose an answer, or because the engine had no document yet, cannot be settled that way, and the next reconcile sent the same document again over a write that had read it. So the changes the write was sent after await the revision the 503 names (State.Awaiting, saved as the answer comes back, in the state Secret's awaited-revisions), and finding that revision running settles them (State.Found, in settleRead), however the pointer reads. Only an activation the engine confirmed whose record is lost while the pointer could not be read still costs one more activation. What a write needs beside that is how its document reads every name, which nothing else says once the render has moved on: the state records it under the write's appliedHash before every write is sent (State.Pending, saved even where no store change is marked; the state Secret holds it under pending-activations), and a revision the engine runs that is a write of the operator's, whose reading the state does not record, records that reading (settlePending, State.Confirmed): a confirmed unconfirmed write, and a confirmed write whose record in the state the reconcile that made it did not get as far as saving, where status.config still names it. The readings the status no longer names an unconfirmed write for go (KeepPending), so the state does not grow with every write. The pending readings sit behind one pointer of the State beside the reading of the revision it knew, which a confirmation makes of one of them (readingRecord), and the awaited revisions beside the changes they settle, so that a State, which callers pass by value, stays inside the size a value parameter is held to: 64 bytes, where it was 72. A reconcile whose document is held does the same against the revision it reads, and leaves status.config and the report to the next one that applies a document. It reads the engine while a write is unconfirmed even where nothing else is owed: returning first, it left the value of a deleted Secret that a running write lets go, and the revision the state records does not, for that write's tool servers to start with until the document applied. Where it activates such a write again for a value it withdrew, the write's record moves on to the revision that activates it (recordReactivation), which the next reconcile then confirms and reports: left on the write, the reload read as a revision somebody else activated, and the write over it as drift corrected. A reload the operator asks for is a write all the same, and the engine answers its lost compare-and-set the same 503. The stored revision may land yet, so it is recorded from the answer, before anything is read back: a read that fails, or finds the revision running while the fleet cannot be read, says nothing else. Where the revision read is the operator's, the stored one is recorded as an unconfirmed write of its document (reloadUnconfirmed), with its reading saved before the reload and the digest of that document (documentSHA256): the engine activates again whichever revision is active when the request lands, so a revision found running another document reloaded one somebody activated in between, and is not the operator's (runsItsDocument). The changes it was asked after await it. The record replaces nothing of its own: its base is the operator's write, and where that write is recorded too, the confirmation of either reports what the write replaced, once (confirmUnconfirmed follows the bases); copied onto the reload, it named the operator's own write as drift. A revision the read back settles as the reactivation is recorded once: recordReactivation drops an unconfirmed entry of it, which the API server refused beside a second one. A confirmation sets a flag of its own (confirmed) rather than wrote, which says this reconcile activated what it rendered: humanSeats read the rendered document for the human seats IdentityClaimed names, which a confirmation of an earlier render, with this reconcile's refused, are not. The in-sync path and the Applied event read either flag, and the flag is reset with every fresh read. The applying concept page, the troubleshooting guide and the status fields' docs say all of this. The engine client tells this answer from a lost race (UnconfirmedActivation). The withdraw tests' storeEngine answers as crewlet#142 does where the tests need it: a write refused on its If-Match (412 revision_advanced, a failed precondition, naming the revision that runs and the base as the header gave it) or If-None-Match (412 already_configured), an engine with no document, and GET /fleet refused as a lease table its coordination store did not answer is (the code's sentence, the condition's own words as the detail, and the five seconds of the health tick); and a write or a reload whose activation's answer is lost as crewlet's main words it, with the node, the base, the hint and the Retry-After it sends; its fleet's activation names the revision that runs, or nothing for an engine with none; a write it is to refuse is refused in put, as its refusal says; and it names each revision past every one it named and every one a test put in its history by hand, as the serving tests do. unconfirmedWrite rotates a value before the write: r1 is the rendered document and lets the withdrawn registry go, so the withdrawal alone is settled by activating r1 again, with no write to lose. TestARevisionTheEngineCannotOpenBlocksTheDocument counts the writes the engine refused on their condition as well as the ones it stored, and TestAnApplyIsRecordedBeforeAnythingIsDeletedAfterIt counts the reading saved before the write among the state's writes. TestAnActivationTheEngineCouldNotConfirmIsNotWrittenTwice, TestAnActivationThatLandsLateIsTheOperatorsOwn, TestAnActivationThatDidNotLandIsSentAgainOnItsCondition and TestASeedTheEngineCouldNotConfirmIsReportedAsTheSeed fail without the fix, as do the record, the unread-fleet and the bound cases; TestAnActivationTheEngineCouldNotConfirmIsNotWrittenTwice also fails with the readings never dropped. TestAnUnconfirmedActivationOtherwiseOvertakenIsNotSentAgain fails with a write built on another revision kept as overtaken, TestAnUnconfirmedActivationIsNeverWrittenOverUnread with the fleet read skipped, and TestUnconfirmedActivationsAreBounded with either list untrimmed or keeping the oldest, or the constant moved off the CRD's maxItems. TestADriftCorrectionTheEngineCouldNotConfirmIsReportedOnceItRuns fails with the drift left off the record or the report left out of the confirmation; TestAConfirmationThatLosesItsRaceIsReadAgain with the lost race not wrapped as one; TestAWriteBuiltOnTheOneConfirmedStaysUnconfirmed with that write dropped; TestAConfirmedWriteIsReportedAppliedOnceTheEngineAppliesIt with the Applied event left to the wrote flag; and TestAConfirmedWriteLeavesTheSeatsToTheRevisionThatRuns with the confirmation setting wrote. TestAWriteConfirmedLateIsHeldToTheDeadlineFromWhenItWasFound fails with the confirmation dated by the write. TestAWriteRecordsItsDocumentsReadingBeforeItIsSent fails with the reading not recorded, or not saved where no store change is marked. TestAHeldReconcileReleasesWhatARunningUnconfirmedWriteLetsGo fails with the held reconcile returning before it reads the engine, with what the confirmed write lets go not read again, with the reading not confirmed, with the reading not recorded before the write, and with the record not moved on to the reload. TestAReloadTheEngineCouldNotConfirmIsTheOperatorsOnceItRuns fails with the stored reload not recorded. TestALandedWriteActivatedAgainStaysInSyncThroughAReconcileThatFails, TestAWriteAReactivationOvertookIsNotSentAgain, TestAHeldDocumentSettlesAnUnconfirmedWriteThatRuns and TestAConfirmationTheStateMissedIsSettledLater hold the activations that follow a write to what its epoch marks settle. In secretsync, TestAPendingReadingRecordsTheRevisionFoundRunningIt fails with a confirmation settling a store change, or dropping a reading, or with KeepPending keeping one it was not given, and TestPendingReadingsSurviveASaveAndALoad with what only a hand edit writes kept. TestAWriteWhoseAnswerIsLostWhileTheFleetIsUnreadableSettlesWhatItRead and TestAFirstWriteWhoseAnswerIsLostIsNotWrittenAgain fail with the write's changes not left awaiting the revision it stored, or with settleRead not settling them on finding it (the fleet answers the read of what every pod runs and not the ones after the changes: an unreadable fleet keeps every value, so nothing would be withdrawn); TestAReloadWhoseAnswerIsLostIsTheOperatorsWhenItsReadBackFails with the stored reload recorded only where the read back found the revision it read, with settleRead not settling what awaits, and with the reload's changes not left awaiting it; TestAReloadOfAnotherRevisionIsNotTheOperators with the document's digest not checked; TestAReloadThatLandsWhileTheNextOneFailsIsRecordedOnce with recordReactivation keeping the entry it records; TestAReloadOfAnUnconfirmedReplacementReportsWhatTheWriteReplaced with the reload keeping the write's replacement, or the confirmation reporting its own base; and TestAConfirmedWriteLeavesNothingOvertaken with the overtaken writes kept. In secretsync, TestARevisionStoredForAnActivationSettlesTheChangesItWasAskedAfter holds Awaiting and Found, the save and load of awaited-revisions, and a later change waiting for its own activation. The engine activates the revision it runs again itself at a sync's write or removal of a value that revision reads, so the operator's own write or reload, whose answer these tests lose, comes only where that activation failed: the tests make it fail (operatorActivates). Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The EngineClass floors stop a new class asking for less than 11Gi, or 14Gi with the files in the stream's own bucket, but a company keeps the claim it was made with (dataVolumeSize reads the StatefulSet's own claim template), and the API server's validation ratcheting lets a class stored at an older floor go on updating, and stamping its claims on new companies. Under crewlet#142 such a volume's stream cannot hold what a fresh node reserves (its nine state logs at their floors, 8120172544 bytes, and with the bucket 10826896725), and Crewlet refuses to boot on it wherever it has a log to create: a fresh node's every log, and the chat log #142 adds to a stream made before it, which a main-era 9Gi or 12Gi volume cannot fit. Nothing named the volume: the operator rolled the pod onto that engine and reported PodsNotReady over a restart loop. config.StoreMaxBytesFloor stays 6 GiB, so the document itself is valid and crewlet validate only advises; what refuses is the broker. The render already refuses, and the controller already holds the pods back on, the start-time refusals it can see coming (an *AddressError, an unacknowledged password-only sign-in). This follows that idiom: Bootstrap refuses a volume whose share is under what a fresh node of the class reserves (freshNodeFloor: the logs alone with the files in S3, the bucket beside them otherwise) with a *VolumeError (ErrVolumeTooSmall) carrying the volume, its stream, that reservation and the least whole GiB volume whose share holds it (leastDataVolume: 11Gi and 14Gi, which TestContractStreamFloor holds to the CRD's floors). The two figures move from the floor test into the render, where they are now read, and the floor test renders its below-floor case at the floor with the smaller stream set by hand, since the render refuses the smaller volume. The controller reports RuntimeReady=False with the new reason DataVolumeTooSmall, keeping the pods and their bootstrap Secret as they are. The message names the volume, the size it needs and what to grow (volumeTooSmallMessage). A company that has a data claim, or a StatefulSet that makes one, is told the claim to grow, and growing it clears the hold once the expansion lands. A new company made from a class stored under an old floor has neither: its volume would be the class's, so it is told to set the class's spec.data.storage.size, at which the operator then creates its volume, since there is no claim to grow. The hold cannot keep running an engine whose image already moved to one that cannot reserve its logs; what it changes is that the cause is named. TestBootstrapRefusesAVolumeUnderTheStreamFloor covers, for each place a class keeps its files, the main-era floors (12Gi and 9Gi), a volume a GiB under the new one, a byte under the least volume whose share holds the reservation, and zero, and that the least such volume and the floors render; it goes red with the render's check off and with the S3 class held to the bucket's figure. A Company runtime envtest spec holds a company whose StatefulSet was made with a 12Gi template back with the reason and its message, its template and bootstrap Secret untouched, then runs it on the 14Gi its claim was grown to. TestAVolumeUnderTheFloorNamesWhatToGrow holds the message to the claim where one exists and to the class where none does, and fails with the class's remedy dropped. The reason is in the troubleshooting table, by both remedies, and the runtime page's list of holds, and the troubleshooting bullet for the broker's refusal now says the operator holds such a company. TestBootstrapBoundsTheStreamByTheVolume's smallest case moves from 12Gi, which the render now refuses, to 14Gi. The reason's comment quotes both figures, so TestStreamFloorFiguresAreTheEngines lists the API's common_types.go with them, under the mirror table's constant for that path (commonTypesGo). Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Unit.channel's doc says what the field holds and how Crewlet spells it, and since crewlet#142 that on Chatlet each channel is a room whose members follow the org chart. It did not say that such a room belongs to one unit: Crewlet refuses a second unit naming the same channel, so a child unit that should share its parent's room names none, which is the one way to share it. The rule over the field itself, and its test against the engine (TestUnitChannelFollowsTheEngine), came with #51. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet now runs a chat of its own, Chatlet: a company that declares no Slack or Mattermost (no integrations block for either, and no seat's Slack app) derives chat.backend chatlet, which is every company the operator renders without such an Integration. Two of its advisories moved with it. The warning of a Human seat with no chat contact is given only on a vendor surface, judged per surface: a seat with no account on the Slack or Mattermost the company talks on, a GitHub, GitLab or Atlassian id not counting. And a company on Chatlet whose units declare no channel is advised at units[0].channel that no team has a room the chart keeps. #51 already expects what the engine says of the guide's company and of the optional-keys case, which names no channel and is advised at units[0].channel, and the guide names its unit's channel. That left nothing holding the narrowed contact advisory, which the Seat API and two guides describe, so a new contract case, TestContractContactAdvisoryFollowsTheChatSurface, holds it on the Slack case: a Human seat with a GitHub login alone is warned at its contact of having no account on Slack, and one with a Slack member id is not. It goes red with either expectation flipped. The Seat contact doc, integrations.md and people.md now say the warning is given only on Slack or Mattermost, which accounts count, and that on Chatlet a person reads the rooms on the dashboard and nothing is warned; the troubleshooting page's list of what the engine says names both advisories, and two test comments stop calling a GitHub login a chat contact. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet now runs a chat of its own, Chatlet: a company whose document declares no Slack or Mattermost derives chat.backend chatlet, so every company the operator renders without such an Integration gets a room per unit channel, its seats and people talking there, and Chatlet's workspace on the dashboard, its messages kept on the chat state log (CREWLET_CHAT_LOG) every data node runs whatever its company's chat. No operator page said so; the integrations guide now does, linking the engine's Chatlet page, and says connecting Slack or Mattermost moves the company onto that vendor's chat. The engineConfig doc's list of the tuning blocks it carries also left out artifacts, which it passes through like the others and which holds no credential; the list names it. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet's company document gains a chat block: backend chatlet, vendor or none, and under chat.chatlet the policy of Chatlet, its own chat: a message horizon, whether a room is private unless its creator says otherwise, and when a seat shows it is working. The tracker and the knowledge base, the two other surfaces a company picks between the engine's own and a vendor's, are typed fields of the Company; chat reached the document only as an engineConfig entry, which nothing validated before the dry run and which the API reference did not describe. spec.chat.backend is Chatlet, Vendor or None with no default: unset, the engine derives it (Vendor beside a Slack or Mattermost Integration or a seat's Slack app, Chatlet otherwise), and a defaulted Chatlet would render beside such an Integration, which the engine refuses. spec.chat.chatlet takes messageRetentionDays (0 for ever, or 30 to 3650), a defaultChannelPrivate and typingStatus (the WorkingStatus the vendor blocks already use), and is refused beside Vendor or None; with no backend it applies wherever the engine derives Chatlet, as Crewlet takes it. Chatlet beside a chat Integration, Vendor without one and a chatlet block with no backend beside one turn on other objects, so they are left to the engine's dry run, as a tracker of Jira without a Jira Integration is; TestContractChatSettingsFollowTheEngine holds that the engine refuses each where it says (chat.backend, chat.backend and chat.chatlet), that the document attributes each to the Company, and that it takes a chatlet block with no backend on a company with no chat Integration, and goes red when any of them is answered otherwise. chat joins the keys engineConfig may not set, beside main's assistant. The new chat company case renders every setting, which TestCompanyGolden, TestContractCompany and TestDiffContract hold. A new test, TestEngineConfigRefusesEveryTypedKey, holds typedKeys to the keys the document type renders and to the CRD's engineConfig rule, which nothing compared, and fails with chat taken out of either. It found the assistant missing from typedKeys, which main's assistant field renders and the CRD refuses in engineConfig: Diff took an assistant block only the engine had for a tuning block the Company leaves to the engine's defaults, rather than a change somebody made there, so typedKeys gains it too. The CRD suite covers each rule's edges. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet now answers "is a person waiting on this seat" with one rule per chat backend (notify.AddressRule), which its working indicator, its delivery check and its prompt all read: a direct message, a personal mention, or a thread the seat is in for a reason that owes an answer, never one it reached through a room-wide address or only by having spoken there. WorkingStatusAddressed still said "a thread the seat follows", which is the second of those exclusions; and since the typed chat block reuses WorkingStatus for Chatlet, it now also says what addresses a seat there: a reply in a thread it started, and a person's message in a unit's room that reached nobody else and so falls to the unit's lead. The enum's values are unchanged; only the API reference moves. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet now answers a write whose If-Match names a revision another
write has moved past with 412 revision_advanced, a failed precondition,
whether the revision was stale when the request arrived or lost the
activation's compare-and-set after it; only a write that named no
revision loses with 409. The operator matches ErrRevisionAdvanced by its
code alone, every write of the document it makes carries an If-Match or
a create-only condition, and the one request of its that names no
revision, its activation of the running revision again (POST
/config/reload), still loses with 409, so it went on working; but the
client's doubles answered their stale-revision case 409, to a write that
sent no If-Match, which only a reload is.
TestErrorMapping's cases may now carry the request they make: the stale
case sends If-Match and is answered 412, and a lost race with no
If-Match is answered 409, both ErrRevisionAdvanced.
TestRevisionAdvancedNamesTheCurrentRevision reads both revisions off the
412. TestEngineStaleIfMatchIsRevisionAdvanced asserts the real engine's
412 too, and fails expecting 409. The doubles' refusal for a missing
grant now carries the engine's own sentence for [admin] ("You don't have
the access this needs (Admin). Ask an admin for it."), which no operator
code reads. The client's comments say which status each write gets,
the reload's 409 among them.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
The runtime page's backup command named the node with -url, which crewlet#142 renames -api on every command that reaches a running node, with no alias, so against it the command stops at "flag provided but not defined: -url". It now names the node with -api, and the sentence before it says which flag that is. The token it pipes in is as it was. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet's bundled tool skills now trigger on slack-mcp-server's tool names under the slack_ prefix as well as on Mattermost's: the mentions, hand-off and directives skills on slack_conversations_add_message, and the channel skill on slack_channels_list, slack_channels_me and slack_users_search too. The guide's Slack MCPServer sets that prefix and said nothing about why, so another prefix would have looked harmless. Say it in the Slack section, linking the engine's own account of the pin. The Mattermost section named only the posting tool and the mentions skill, while the channel skill already triggered on three more of that server's tools and the mentions skill on its edit; name all five. Every name was read in the pinned releases, 1.3.0 and 0.6.1. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
Crewlet answers a PUT /config whose activation it sent to the fleet's coordination store, and never heard back about, with 503 unavailable, "outcome": "unknown", the stored_revision_id and the node that stored it, under the Retry-After of an undecided answer, with a hint saying how to look (RefuseActivationUnknown, as crewlet's main words it): the fleet may run that revision or may not, and only a read of the fleet's activation says which. #142 also answers a condition of the node on /config, /setup and /secrets (a coordination store that did not answer, its store busy with other writes or closed for a snapshot, a deadline) as 503 unavailable with the condition in words, and a Retry-After where waiting clears it. What the operator does with the unconfirmed activation is the earlier commit's, fix(controller): never write twice over an activation left unconfirmed: engine.UnconfirmedActivation reads the answer, the write is recorded in status.config.unconfirmed, and the next reconcile reads the fleet's activation before it writes anything, sending the write again on its first condition only while the fleet still runs the revision it was built on. That commit wrote the troubleshooting guide's account and the reconcile's. What the client still said was out of date: its package comment and ErrUnavailable called every 503 a request the engine asks to be sent again, and ApplyConfig named no such answer. Read that way, this answer says to send the same write again at once, which loses the race to its own first attempt once that one did activate. So ErrUnavailable says what a 503 is and that UnconfirmedActivation tells this one from the rest, ApplyConfig names it and the read that settles it, StoredRevisionID names both answers that carry a stored revision, and the package comment calls ErrUnavailable a request the engine could not answer now. TestUnconfirmedActivationIsTheEnginesUnknownOutcome also decodes the engine's answer: ErrUnavailable and not ErrRevisionAdvanced, the undecided answer's two-second wait, and the engine's hint, which names the node whose history alone holds the revision. It fails with the Retry-After or the hint no longer read. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 now words a 503 by its Retry-After. Under one, unavailable
and identity_unavailable keep the code's own sentence ("Try again in a
moment"). Under none, which the engine writes for a condition no wait
clears (a record a node can't read yet, a node the fleet evicted), they
say "This can't be done here, and waiting won't change that." and "Can't
check who you are here, and waiting won't change that.". A write whose
outcome is unknown keeps the code's sentence, so the unconfirmed
activation's double is still the engine's answer. And a condition of the
node on /config, /setup or /secrets (its store busy with other writes,
closed while it adopts a peer's snapshot, a deadline) is answered 503
unavailable with the condition in words as the detail, where crewlet's
main answered 500, or, for an activation pointer it could not read, 503
with a reason field.
The operator goes on doing the right thing: a 503 is ErrUnavailable
whatever it says, and the reconcile asks again every ten seconds, or
sooner where the engine said when, since nothing else tells it that
somebody has cleared the condition (an unconfirmed activation, which
UnconfirmedActivation tells apart, is settled by a read before anything
is written). But five doubles answered a 503 the engine no longer
writes, none answered the 503 a condition of the node now is, and the
client's and the guide's account of a 503 had no word for one no wait
clears.
- TestErrorMapping's engine that could not decide answers as the
authority table's unknown does, with a two-second Retry-After, and a
new case answers the identity directory a write that removes a human
seat reads, refused on this node until somebody acts: no Retry-After,
and the no-wait sentence.
- TestUnavailableCarriesTheEnginesRetryAfter gives each case the body
the engine writes beside its header, the no-wait sentence where there
is none.
- TestFleetRefusals's unreadable lease table now carries the condition's
own words as the detail, which #142 writes beside the code's sentence
and the five seconds.
- TestStartingWriteIsAskedAgainWhenTheEngineSays, and the starting 503
the controller's quoting tests end on, give the detail #142 writes for
a node recovering what it holds, which names its store, its broker and
its copy of the estate in parentheses where crewlet's main sets them
off with dashes.
- TestUnconfirmedActivationIsTheEnginesUnknownOutcome's engine that
could not store the write answers as a coordination store that did not
answer does: two seconds, and the condition's own words as the detail.
With no Retry-After, its message was the one the engine no longer
writes beside that body.
- TestActiveConfigWhileTheFleetCannotBeRead's unreadable pointer answers
as #142 answers a condition on a read: the five seconds of its health
tick, and the condition's own words as the detail, which the client
takes for the error's message, where it carried main's reason and a
message the engine never wrote.
- TestPutSecretErrorsNeverContainTheValue gains a store too busy to take
the value, answered 503 with the condition in words, beside the fault
of the engine's own it still answers 500.
ErrUnavailable's comment says what a 503 with no RetryAfter means, and
why the client retries it the same way. troubleshooting.md's
EngineUnavailable row and its causes name the 503 no wait clears, and
the 503 of a condition of the node that clears, which the message
quotes, and say the operator keeps asking until it is cleared in the
engine.
TestErrorMapping's new case fails with ErrUnavailable matching every 503
but identity_unavailable, and
TestUnavailableCarriesTheEnginesRetryAfter's "none" case fails with a
missing Retry-After read as a second, and
TestActiveConfigWhileTheFleetCannotBeRead with the engine's message
taken over its detail.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142's engine now refuses a write that removes a human seat,
when the node can't read who holds it, under a code of its own:
503 seat_holders_unavailable. It used to be identity_unavailable, whose
sentence ("Can't check who you are") is about the caller. The answer
also carries its words in `detail`, not `hint`: the identity log's own
sentence where the log refused the read, and otherwise that nothing was
written and to try again in a moment. Its seat_held hint lost its em
dashes too.
The client's doubles still answered with the old bodies, so they no
longer described anything the engine writes:
- TestErrorMapping's "a condition no wait clears" now gives the engine's
seat_holders_unavailable body for a node the fleet evicted. It has no
Retry-After, the code's no-wait message, and the eviction sentence as
its detail.
- TestUnavailableCarriesTheEnginesRetryAfter gives each case the body
that matches its header, the two-second answer included.
- TestSeatHeldNamesEachHolderButNotTheirLogin's hint is the engine's.
The operator's behaviour holds. Any 503 is ErrUnavailable whatever its
code, so the reconcile asks again every ten seconds, or sooner where the
engine says when. ErrUnavailable's comment now names the new code among
the 503s it covers.
Against that engine the package passes. With ErrUnavailable refusing
seat_holders_unavailable, TestErrorMapping's case and three of
TestUnavailableCarriesTheEnginesRetryAfter's fail.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 answers a condition no wait clears (a record a node can't read yet, a node the fleet evicted) with 503 under a code of its own and no Retry-After, and words it so: "This can't be done here, and waiting won't change that." The operator still wrote ConfigApplied=False EngineUnavailable as "The operator's request to the company's engine failed, and it tries again in 10s: engine answered 503 unavailable: This can't be done here, and waiting won't change that.", its own half promising the retry the engine's half says changes nothing, and its SecretWithdrawalFailed event did the same. The client now keeps a 503's outcome (APIError.Outcome) and tells this answer apart (Waitless): a 503 under a code of the engine's, with no Retry-After, that is not about a write whose outcome the engine does not know, which may have landed and whose answer says to try it again. The condition and the event then say that the engine answered that waiting won't clear what it met, so somebody has to clear it in the engine, and that the operator asks again every ten seconds until then, since nothing else tells it the condition has cleared. A 503 under a Retry-After keeps its promise of a retry when the engine said. The troubleshooting guide says the condition words it so. TestWaitlessIsTheEnginesNoWait503 holds Waitless to the engine's answers: the no-wait 503 under unavailable and under seat_holders_unavailable, and neither a 503 under a Retry-After, one whose write may have landed, a proxy's nor a 409. It fails with the outcome left unread. TestANoWait503PromisesNoRetry holds the condition and the event to it, and fails with the condition's wording dropped. TestAConditionThatQuotesAnErrorEndsItOnce gains the no-wait 503. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 words its 409 unreadable_revision and unsealed_revision for
a person. The detail says what happened ("This node's keyring doesn't
hold the key that revision is sealed under."), where crewlet's main
quotes the keyring's own error ("secrets: open document: secret could
not be decrypted."). An unsealed revision's hint names the write that
replaces it ("... to replace it, an admin sends the whole document from
your own copy with PUT /config, or imports it with
`crewlet config import`, either of which stores it sealed."), where
main's has it imported again; a lost key's hint is main's. The code,
the message, the revision and the missing Retry-After are as they were.
The client's doubles (TestErrorMapping,
TestAnUnreadableRevisionNamesItAndItsRemedy) and the controller's
(TestARevisionTheEngineCannotOpenBlocksTheDocument) still gave main's
bodies, so they described nothing this engine writes. They now give
its details and its unsealed hint. The operator's behaviour holds: the
message is read as the detail whatever it says.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 reads the active revision off the fleet's activation pointer, and answers one this node's keyring does not open under two codes of its own: 409 unreadable_fleet_revision when the fleet sealed it under a key this node was never given, and 409 unsealed_fleet_revision when it arrived without a seal, which every engine refuses since every engine seals what it activates. Both name the revision, say what happened in their detail and the remedy in their hint, and carry no Retry-After, as unreadable_revision and unsealed_revision do. The client matched only those two, so the fleet's codes went back to being an engine the operator could not reach: ConfigApplied False EngineUnavailable, a promised retry and a look every ten seconds. Now ErrRevisionUnreadable matches all four, and RevisionUnreadable says what restores each. For unreadable_fleet_revision, every pod has its keyring from Secret <company>-keyring, so the data pod lacks a key the Secret gained after it started and takes it when it restarts, or the key has gone from the Secret and is put back from the backup. For unsealed_fleet_revision, the revision came from outside the fleet, and the engine's hint says what to do. The troubleshooting guide names the four codes, and the engine's config_revision_unreadable log line. TestErrorMapping, TestAnUnreadableRevisionNamesItAndItsRemedy and TestARevisionTheEngineCannotOpenBlocksTheDocument gain the fleet's two answers as #142 words and ends them. They fail with the two codes left out of the sentinel, and the controller's cases with the two remedies dropped. Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 now opens its advisory on a Human seat with no account on
the company's chat vendor with the seat's handle quoted: 'the human seat
"jane" has no account on Slack', where it said "jane is a human seat
with no account on Slack". A reader that puts the engine's sentence in
sentence case wrote the handle that opened it as a seat nobody declared
("Jane").
TestContractContactAdvisoryFollowsTheChatSurface holds the advisory to a
substring of what `crewlet validate` warns, so against that engine its
"a GitHub login alone" case found no advisory it expected and one it did
not. It now expects the engine's words. The guides and the Seat API
describe the advisory without quoting it, and hold.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 ("fix(api): let PUT /config replace an unsealed active
revision") tells apart three seals it cannot open where it told two. A
payload with no seal at all is unsealed_revision or
unsealed_fleet_revision as before; one wrapped as a sealed document in a
form the engine's build does not read, most likely a newer build's, now
answers 409 unknown_seal_revision, one code for either origin, naming
the revision, with no Retry-After, and a detail and hint of its own:
"This node's copy of that revision is sealed in a form this build of
Crewlet doesn't read." and "A newer build of Crewlet most likely sealed
it here, and no write here replaces it, since that would discard it
unread; run that build on this node again." (for the fleet's revision,
"...; upgrade this node to the build the rest of the fleet runs.").
The client did not know the code, so that answer read as an engine the
operator could not reach: ConfigApplied False EngineUnavailable, a
promised retry and a look every ten seconds, over a revision no wait
opens. ErrRevisionUnreadable now matches it, and RevisionUnreadable
gives it its own account and remedy: the keyring Secret is not at fault
and no write replaces the revision; the data pods run the build that
sealed it, the engine image of the Company's EngineClass (with
spec.runtime.version as its tag where the class allows that), and a
data pod still rolling to it takes it when it restarts. Its lead says
the engine does not read the revision's seal, where every other cause
says its keyring does not open it, which would send the reader to the
keyring.
TestErrorMapping, TestAnUnreadableRevisionNamesItAndItsRemedy and
TestARevisionTheEngineCannotOpenBlocksTheDocument gain both origins of
unknown_seal_revision as the engine words them. They fail with the code
left out of the sentinel, and the controller's with the remedy dropped
(the keyring's words then). The troubleshooting guide names the fifth
code and what fixes it.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
crewlet#142 ("fix(api): let PUT /config replace an unsealed active
revision") lets a whole document replace an active revision stored
without a seal: PUT /config, dry runs included, over one that answers
409 unsealed_revision or unsealed_fleet_revision goes on and reads
nothing from it, judged as a document replacing nothing, while If-Match
is still judged against that revision and a dry run names it as its
base. Its hint for both codes names exactly that write. Every other
write is still refused over it, a reload included, and a revision sealed
under a key the keyring lacks, or in a form the build does not read,
stays refused for every write, PUT included.
The operator blocked the Company on both codes as on the others,
RevisionUnreadable saying the engine "neither shows the document nor
takes another", so under Enforce a revision the engine's own remedy
replaces stayed until somebody stopped the data pod. Now, under Enforce,
converge takes the refusal of the read (engine.Unsealed, which answers
for the two codes and a named revision only) and goes on as over an
engine with no document but for the write's base: the rendered document
is written conditioned on that revision. The new revision's summary,
ConfigApplied and a Warning RevisionUnreadable event say which revision
it replaced, as the engine's active revision, whose nodes ran the one
before it. Under Seed, which writes only into an engine with no
document, and Observe, which writes nothing, it still blocks, and the
remedy says what replaces it: an admin's whole document, as the engine's
hint says, or the Enforce policy. unreadable_* and unknown_seal_revision
block under every policy as before. A revision without a seal that lands
after the reconcile's read, the one way it is still reported under
Enforce, is looked at again ten seconds on rather than at the resync
interval, since the operator's own next look replaces it.
Nothing the secret store holds is deleted before that write. Which names
such a revision references is what nobody can read, and it is not what
the engine's nodes run: none adopts a body it cannot open, so each runs,
and boots again when it restarts, the revision it applied before,
reading the store as it finds it. A value deleted while the replacement
is held (the engine refusing it, a seat somebody holds, or a write not
sent yet) is one such a node may start without. secretsync gains
Desired.KeepListed: every owned name the store lists is kept, an absent
or released one included, whatever the state records of the revision's
reading, while every value is still pushed, and Holders names that
revision as what keeps each absent name the store retained, although the
listing is Sync's and Withdraw's alone, so that SecretKept and the
conditions say why; enforce keeps that way before the write over such a
revision, and so does the withdrawal of a document held for a Secret,
which read the 409 and reported a SecretWithdrawalFailed at every look.
The sync after the write decides by the replacement as after any write,
and activates it again for the store it lets go. SecretKept,
SecretsSynced and a part's ResolvedRefs line say why a value is kept for
it: the revision, which they call the engine's active revision and never
the one it runs, is stored without a seal and nobody can read it, while
the nodes run the one before it. That revision, the one each node
reports running on GET /fleet, is read by its id (GET
/config/revisions/{id}) and its names kept beside the listing until the
nodes run the replacement, as for any revision a node runs other than
the fleet's (keepServing).
No activation is asked for over such a revision: the engine neither
shows nor activates it again, so a reload is refused every time. Asked
for, a document the engine refused, or one held for a Secret, came back
every ten seconds over a refusal no retry clears, each time with a
SecretWithdrawalFailed. A store change owed, a withdrawal or a value
written, stays owed, and the replacement's activation settles it. The
revision without a seal settles nothing the store owes, although the
fleet's pointer names it at an epoch above every mark: its nodes run the
revision before it, which may have activated before the change. That
takes nothing new: a store change is settled only by the revision GET
/config answers with (settleRead), and over a revision without a seal it
answers with none.
A seat_held over the replacement is worded as the engine asks: the
document does not keep those human seats, since nobody can read whether
the revision it replaces had them. A write over the revision the engine
could not confirm is settled against it, so it is kept while it may
still land, records that it replaces a revision without a seal
(replaced: Unsealed), says so in its EngineUnavailable message, and is
reported once a read shows it runs as a confirmed replacement is, its
ConfigApplied and Warning RevisionUnreadable included (reportReplaced).
A reload of that write whose answer is lost too replaces nothing of its
own (reloadUnconfirmed): its confirmation reports the revision without
a seal replaced, once, and never the operator's write as one.
The write is now conditioned on the revision the reconcile read, not on
the one the dry run names. They differ only when another writer
activated a revision in between, and conditioned on the dry run's, the
write went over that revision unseen, reported as drift from the one
read, with the names the sync kept decided for that one; over an
unsealed revision it would have said it replaced that revision. Now the
engine refuses the write before storing anything and the reconcile
starts over from a fresh read. Seed's override to create-only goes with
it: the read found no document wherever Seed writes. With it goes the
way TestAWriteBuiltOnTheOneConfirmedStaysUnconfirmed reached a write
built on a revision no read had found running, the dry run's base, and
that test with it. One way stays, and a confirmation still keeps such a
write unconfirmed, now saying why: a reconcile that holds its document
reads the engine without recording what it finds, so it may activate a
running unconfirmed write again for a value it withdrew, and lose that
reload's answer too (reloadUnconfirmed). ApplyConfig's doc says what the
engine does with If-Match over such a revision: judged against it as
against any other, and a write without one replaces it unconditionally.
The storeEngine double refuses what reads a revision it cannot open, per
revision, as the engine does: a read of the document or of the revision
by its id, a reload, and a write (dry runs included) over any but one
without a seal; and it logs the requests it answers. Its reload moves to
a helper of its own (reload), which keeps serve under the cyclomatic
bound. A test that needs the operator to write over r1, which is the
rendered document and read every value, rotates a value first. The
seat_held doubles, the client's and the storeEngine's, word the detail
as #142 does ("that somebody still holds", the seats joined as a list,
which the operator does not read), and the storeEngine's carries #142's
hint and message beside it. Tests:
TestEnforceReplacesARevisionStoredWithoutASeal (both origins: one write,
If-Match on the revision, the deleted Secret's value still there when it
is written, gone after with the replacement activated again, Applied
once the epoch applies, summary and event, nothing written the next
time), TestAReplacementOverARevisionWithoutASealKeepsWhatTheNodesRun
(the nodes on the revision before it, read by its id, its names kept
after the write until they run the replacement),
TestAReplacementRefusedOverARevisionWithoutASealKeepsTheStore (a name
only the nodes' revision may read kept, the activation owed left owed,
no read or reload after the dry run, no SecretWithdrawalFailed, and both
settled by the replacement once the document is valid),
TestAHeldDocumentOverARevisionWithoutASealKeepsTheStore (and what
SecretsSynced and ResolvedRefs say of it),
TestAHeldDocumentOverARevisionWithoutASealOwesItsActivation (an
activation owed left owed although the pointer names the revision at an
epoch above its mark, with no reload asked for and no
SecretWithdrawalFailed),
TestUnderEnforceAnUnsealedRevisionAfterTheWriteIsLookedAtSoon,
TestARevisionTheEngineCannotOpenBlocksTheDocument (the sealed causes
under Enforce and Seed, both unsealed origins under Seed and Observe,
nothing written and the store untouched),
TestAWriteIsConditionedOnTheRevisionTheReconcileRead,
TestAReplacementTheEngineCouldNotConfirmIsKept (now its record, its
EngineUnavailable message, and its report once it lands),
TestASeatHeldOverAReplacementSaysTheDocumentDoesNotKeepIt,
TestUnsealedIsTheOneRevisionAWriteReplaces,
TestAReloadBuiltOnAnUnconfirmedWriteStaysUnconfirmedOnceTheWriteIs,
TestAReloadOfAnUnconfirmedReplacementReportsWhatTheWriteReplaced (two
cases more, where the revision the write replaced has no seal), and
secretsync's TestKeepListedDeletesNothingTheStoreLists (and Holders
naming the revision). Each went red with its guard broken: the
replacement removed, allowed under every policy, opened to every
unreadable code, deleting before the write, the held document's
withdrawal or a recorded reading deleting, a store change settled by it,
a reload asked for over the revision by a refused or a held document,
the revision called the one the engine runs (or, in the event, ran), the
look at the resync interval, the kept value's reason, Holders without
the revision, what the nodes run left unread, conditioned on the dry
run's base, unconditioned, settled against no revision, the record,
message or report of the unconfirmed one, without its message, summary
or event, the seat wording, the policy remedies, a write built on the
confirmed one dropped or the held reload not recorded, and Unsealed
taking unknown_seal_revision or a refusal naming no revision.
The troubleshooting guide, the applying concept page and the management
policies guide say what replaces such a revision under each policy, what
the store keeps until then, and how a replacement the engine could not
confirm is reported; the SecretsSynced and ResolvedRefs rows of both
condition tables, and the tool servers guide, name the third reason a
value is kept beside the other two.
The write is conditioned on the engine's own activation of the revision
read where the sync made one (the engine activates the revision it runs
again at a write or removal of a value it reads), once read back as the
same document (syncActivation, split out of reloadedBySync, which
rendersAs now completes): conditioned on the read before the sync, every
rotation that came with a change of the document was refused as a lost
race and written by a second reconcile.
TestAWriteAfterTheEnginesActivationAtTheSyncIsConditionedOnIt fails with
the condition taken from the read. A rotation whose activation another
writer overtook is now refused rather than written over unseen, and
written on that writer's revision by the next reconcile
(TestARotationAnotherWriteOvertookIsNotWrittenOver, out of the table of
TestARotationIsAppliedWhereTheEngineDidNotReactivate).
The engine double refuses a write or a removal in the store over a
revision it cannot open, as the engine does: it asks which paths of the
fleet's revision read each name first (secretsapi, config.Reads), which
opens the revision, and fails the request whole. The tests that pin the
operator's own activation make the engine's fail (operatorActivates),
and TestEnforceReplacesARevisionStoredWithoutASeal expects the
replacement activated again by the engine at the removal after it.
Signed-off-by: Ali Yousefian <aliyousefiaan@gmail.com>
aliyousefiaan
force-pushed
the
follow-crewlet-142
branch
from
October 30, 2026 23:54
b8ddd63 to
29c3486
Compare
8 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Follows crewlet/crewlet#142, which added Chatlet (the engine's own chat, with threads and DMs) and landed on crewlet's
mainas 61d9acdeefd8, together with fixes to the operator found while proving it. The operator's suites run crewlet'smainimage, so they need this from now on.What #142 needs (19 commits). #51 already brought the data volume floors (11Gi with S3, 14Gi with NATS), the nine state logs in a data pod's startup window, the unit channel pattern and the mirror rows #142 moved (
chatLogRead), so the commits of this branch that did the same are gone from it. What remains:spec.chat, which engineConfig may no longer set, andtypedKeysnow lists every top-level key the document renders,assistantincluded); a company with no chat Integration runs Chatlet; a unit's channel names one unit's room; the render expects what the engine now advises a company on Chatlet.DataVolumeTooSmall), naming what to grow, instead of crash-looping.status.config.unconfirmed), settled by the next read, and never written twice. A revision stored without a seal is replaced under Enforce, and one sealed by a newer engine build is reported as that.The operator's own fixes (17 commits, first, each standing on its own), among them:
make testgets a timeout the controller suite fits in.Rebased onto #49 (a sync is one activation): the engine now activates the running revision itself at a sync's write or removal of a value it reads. These commits take that activation where it landed and make their own only where it failed. The tests that pin the operator's own activation make the engine's fail (
failedReload,operatorActivates). The write is conditioned on the engine's activation at the sync, once read back as the same document (syncActivation), rather than on the revision read before the sync, which would refuse every rotation that came with a document change as a lost race.Engine defect found on the way: on crewlet's
main, a secret write or removal fails whole (500) while the active revision is one the node cannot open, because the engine first asks which paths of that revision read the name (config.Reads). The engine double here now answers as the engine does. The consequence: a value that must be written while the engine runs a revision stored without a seal blocks the Enforce write that replaces it. The fix belongs in crewlet (store the value, and answer that nothing was re-activated, as the engine never activates such a revision again) and is not part of this pull request.Checklist
make checkpasses at the head against crewlet bc8e6179a162make manifests generate chart api-docsdocs/)type(scope): summaryProved against crewlet bc8e6179a162, its
mainnow (CREWLET_VERSION=bc8e6179a1):make checkpasses at the head, the controller's envtest suite and the engine suites included, and every commit passes gofmt, tidiness, vet, golangci-lint, the build and the generated-files check. Rebased onto #51, which follows crewlet#478; where #51 had done what a commit of this branch did, the commit is gone or keeps only what #51 does not. Not run here: the e2e suite (make test-e2e, a kind cluster) and the image build.🤖 Generated with Claude Code