Document self-hosted Namespace failover behavior - #5286
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📖 Docs PR preview links
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4469dbf2de
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - If the destination Temporal Service receives the update, both Temporal Services can temporarily consider the | ||
| Namespace active. |
There was a problem hiding this comment.
When readers start from docs/encyclopedia/namespaces/global-namespaces.mdx, lines 18–23 still send them directly to operator namespace update and state that only one Cluster is active at any time. That conflicts with this new guidance: a direct update is now described as forced-only and can temporarily make both Services active. Update that page to point planned failovers to this handover procedure and qualify the single-active statement; otherwise operators can follow the old link and enter the rollback and duplicate-effect window this section is intended to avoid.
AGENTS.md reference: AGENTS.md:L236-L242
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🟡 Changes recommended
The new encyclopedia text introduces “direct failover update” without defining the triggering action, and the self-hosted page’s heading hierarchy/collateral linking needs follow-up for clarity and accessibility.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR expands the self-hosted Multi-Cluster Replication documentation to describe planned vs forced Namespace failover behavior, including how replication/cache behavior affects correctness and why Activities must remain idempotent.
Changes:
- Adds a planned failover procedure using the
namespace-handoverWorkflow and contrasts it with forced failover viatemporal operator namespace update. - Documents how the receiving Temporal Service applies failover updates first and why Namespace cache refresh does not extend the stale-task correctness window.
- Adds guidance on preventing duplicate external side effects after failover (idempotent Activities).
File summaries
| File | Description |
|---|---|
| docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx | Adds the self-hosted operational failover procedure, forced failover behavior explanation, and idempotency guidance. |
| docs/encyclopedia/temporal-service/multi-cluster-replication.mdx | Clarifies metadata replication vs cache refresh semantics and links to the self-hosted operational guidance. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ## How to fail over a Global Namespace | ||
|
|
||
| For a planned failover, run the built-in `namespace-handover` Workflow against the currently active Temporal Service. | ||
| The handover Workflow: | ||
|
|
||
| 1. Waits for the destination Temporal Service to approach the configured replication-lag thresholds. | ||
| 2. Puts the Namespace into the `handover` replication state, which pauses Workflow mutations and task processing. | ||
| 3. Waits for the destination Temporal Service to acknowledge every replication task created before the pause. | ||
| 4. Makes the destination Temporal Service active and restores the Namespace to the `normal` replication state. | ||
|
|
||
| This process creates a brief availability gap while traffic is paused, but it prevents Workflow progress from rolling | ||
| back at the cutover. If the destination cannot catch up during the handover timeout, the Workflow restores the `normal` | ||
| replication state without changing the active Temporal Service. | ||
|
|
||
| Connect Temporal CLI to the currently active Temporal Service, then run the handover Workflow in the `temporal-system` | ||
| Namespace. Use a unique Workflow Id for each handover: | ||
|
|
| #### Namespace versions | ||
|
|
||
| A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| For this reason, send an emergency failover to the destination when the source is unavailable. When both Temporal | ||
| Services are available, use the handover Workflow instead of choosing either side for a forced failover. |
Point Global Namespace failover guidance at the new handover/forced failover procedure instead of the forced-only CLI update, and qualify the single-active-Cluster claim to match. Also drop a sentence in the self-hosted guide that repeated guidance already stated above it.
* Document self-hosted Namespace failover behavior * Update multi-cluster replication documentation Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix global-namespaces contradiction and trim redundant sentence Point Global Namespace failover guidance at the new handover/forced failover procedure instead of the forced-only CLI update, and qualify the single-active-Cluster claim to match. Also drop a sentence in the self-hosted guide that repeated guidance already stated above it. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Duncan Mackenzie <duncan.mackenzie@temporal.io>
Summary
namespace-handoverWorkflow.Why
Resolve the recurring self-hosted Multi-Cluster Replication questions captured in this internal Slack thread.
Validation
vale --config .vale-ci.ini docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx docs/encyclopedia/temporal-service/multi-cluster-replication.mdxyarn build