docs(evm): staging migration plan - #1372
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an executable runbook for migrating the staging CCV DON from Chainlink-node (“CL mode”) to standalone verifier/executor processes, intended to be followed node-by-node in the live staging_testnet environment.
Changes:
- Introduces a detailed, step-by-step staging migration plan covering prerequisites, per-node cutover steps, validation, rollback notes, and open questions.
- Documents required staging-specific operational conventions (kube contexts, IssueOps deploy flow, durable pipeline usage, per-node secrets/DB provisioning).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| cd $CLD | ||
| go run ./domains/ccv/cmd durable-pipeline list --environment staging_testnet | ||
| go run ./domains/ccv/cmd domain config local create --domain ccv --env staging_testnet | ||
| # fill in the JD Cognito credentials it scaffolds |
There was a problem hiding this comment.
Out of curiosity, how do we get these?
There was a problem hiding this comment.
fixed this. five offchain.job_distributor.auth fields; CI has them as OFFCHAIN_JD_AUTH_*_CCV_STAGING_TESTNET, no self-serve path for a local copy, so we'll have to ask #cld-guardian-support or use /run-pipelines. Also fixed the command, it's in cmd/cld.
| Two cautions. First, no archived input in the repo has used | ||
| `consolidateAggregators`/`revokeOrphanedJobs` yet (the payload keys come from the resolver struct | ||
| fields, matched case-insensitively), so treat the first dry run as the test of the spelling. | ||
| Second, node 0 is also the sole member of the `secondary` committee, so it carries a |
There was a problem hiding this comment.
Interesting, I didn't even know we had a secondary committee in staging.
There was a problem hiding this comment.
yeah, it's in topology.toml + on chain + a live job on node 0; details moved to open question 1. Its aggregators have no release in ccv-deploy though, so retiring looks like the cheaper answer.
| Build and publish the standalone verifier and executor images to the internal ECR from a | ||
| chainlink-ccv commit that includes the 20 Aug migration tooling (merge `tt/stagingPrep` first). | ||
| The existing `manual-build.yaml` workflow in chainlink-ccv publishes to the private ECR the stage | ||
| cluster pulls from. |
There was a problem hiding this comment.
We currently push a commit sha tagged image on each merge into main, and then a v0.x.y tagged image every time a release branch is merged. The former is more for speed, the latter for real releases. So I don't think you need to manually build anything, you can just use the commit sha tagged images that have the changes you need, or we can create v0.5.0
There was a problem hiding this comment.
rewritten to use the -rc images from main with v0.5.0 as the alternative
| zero deploy instances; the verifier instances can point straight at it. No executor chart exists | ||
| anywhere, so create `deploy/charts/standalone-executor` with `committee-verifier-base` as the |
There was a problem hiding this comment.
Can you check how solana does it in chainlink-ccv-non-evm-deploy? IIRC they've been working on these standard charts. They may have one for the executor we can use verbatim, as they're also standalone from day 1.
There was a problem hiding this comment.
good call. executor-solana is generic apart from image default/name. Plan now copies it as executor-evm.
| databaseClusterName: chainlink-ccv-dons | ||
| databaseName: ccv_sa_1_verifier_bootstrap | ||
| connectionSecret: | ||
| name: ccv-sa-1-verifier-bootstrap-database-connection |
There was a problem hiding this comment.
Check that you have admin perms to create these secrets - its an Okta request if not (ask me and I can point you to what to request)
There was a problem hiding this comment.
is the only way to check actually running the command? Or can I check the assignment in okta first? I can add details to the doc in case I'm not the one running this
| ```yaml | ||
| databases: | ||
| ccv-sa-1-verifier-bootstrap: | ||
| databaseClusterName: chainlink-ccv-dons |
There was a problem hiding this comment.
I think we should create a new database cluster for the standalones just in case we had to do some manual nuking of stuff and for better isolation, wdyt?
There was a problem hiding this comment.
yep agreed. Adopted as chainlink-ccv-evm-standalone with instance class is a placeholder
| (the image-sync-check workflow errors without it). Shape: | ||
|
|
||
| ```yaml | ||
| - name: ccv-standalone-verifier-1 |
There was a problem hiding this comment.
Small nit: lets have evm in the name somewhere, eventually we'll have all families
| ### P9. Funding source and balance alerts | ||
|
|
||
| Identify what funds staging node accounts today (open question 2) and wire external balance | ||
| alerts for each new transmitter address on the five testnets. The standalone executor does not run | ||
| the node's balance monitor; an unfunded account surfaces as failed broadcasts, not as an alert. |
There was a problem hiding this comment.
Do we have these right now? If not we can probably exclude
There was a problem hiding this comment.
we actually don't. dropped it
| ### 1. Preflight | ||
|
|
||
| ```sh | ||
| kubectl --context $CTX -n $NS exec deploy/$CL -- chainlink admin login --file /v2Secret/.api |
There was a problem hiding this comment.
Is the file /v2Secret/.api in the container itself?
There was a problem hiding this comment.
yeah the chart mounts it at /v2Secret and boots with -a /v2Secret/.api. added a note to step 1
|
Code coverage report:
|
makramkd
left a comment
makramkd
left a comment
There was a problem hiding this comment.
nit: can you change PR title to docs since there's no new code here
tt-cll
changed the title
Description
Testing
Checklist
changelogdirectory)