You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #578 (commit 3de66ab14e3ad0172d686142a58f0a9420ca6e8d) changed parts of the TOML configuration handling from full replacement to deep merge. The follow-up fix in #602 will also preserve typed map fields from user configuration.
As a result, any field in an existing cluster configuration that is not overridden by the operator but is preserved by the new merge behavior may appear in the final ConfigMap and change its content hash. Upgrading the operator may therefore trigger rolling updates of the affected MO workloads even when the user has not changed the CR.
Current behavior
Configuration changes are propagated into the PodTemplate through the following mechanisms:
OperatorVersion < 1.3: the configuration digest is included in the ConfigMap name, which changes the Pod volume reference.
OperatorVersion >= 1.3: the configuration digest is included in the configuration file key and written to the PodTemplate annotation matrixorigin.io/config-suffix.
Therefore, a component is rolled whenever its final rendered configuration changes. Potentially affected MO components include:
LogSet
DNSet / TNSet
CNSet / CNPool
ProxySet
WebUI, the operator itself, backup, and other components that do not consume this MO configuration should not restart solely because of this hash change.
No globally enforced rollout order across components
The MatrixOneCluster parent controller synchronizes LogSet, DNSet, CNSet, and ProxySet in source-code order, but it does not wait for one component rollout to finish before updating the next component. The child controllers also reconcile independently and concurrently. In particular, after an operator upgrade and restart, several components may begin rolling at the same time.
The rollout must therefore not rely on a global sequence such as Log first, then DN, then CN, and finally Proxy. Their rollout windows may overlap.
Rollout behavior within each workload
LogSet and DNSet use OpenKruise Advanced StatefulSet. By default, Pods are updated from the highest ordinal to the lowest ordinal, maxUnavailable is 1, and the update policy is InPlaceIfPossible.
A regular CNSet uses OpenKruise CloneSet. maxUnavailable and maxSurge are controlled by the CN configuration. If not explicitly configured, maxUnavailable defaults to approximately 20%, and there is no stable Pod-name update order.
CNPool currently sets maxUnavailable=1 and maxSurge=0 explicitly.
ProxySet uses OpenKruise CloneSet. If not explicitly configured, maxUnavailable defaults to approximately 20%, and there is no stable Pod-name update order.
When in-place update requirements are satisfied, the container is restarted in place. Otherwise, the update may fall back to Pod recreation.
Each CN group has its own CloneSet and availability budget, so multiple CN groups may also roll concurrently.
Change in fileservice ordering
After deep merge, operator-managed fileservice entries are placed first and unmatched user entries are appended. The previous behavior usually emitted only the LOCAL, SHARED, and ETL entries.
MO currently looks up fileservices primarily by name rather than by array position, but it initializes every additional entry. Correctness risks related to stale fields and extra entries are tracked in #605. This issue focuses on the rollout scope and release safety of the operator upgrade.
Risks
An operator upgrade may roll one or more MO components even when the CR has not changed.
Rolling updates of Log, DN/TN, multiple CN groups, and Proxy may overlap and increase availability risk.
Without examining representative tenant configurations, the upgrade impact and required maintenance window cannot be estimated reliably.
Proposed validation
Select representative real tenant CRs from a development environment, including historical fileservice configuration, custom cache fields, and additional fileservice entries.
Render the Log, DN/TN, CN, and Proxy ConfigMaps with both the pre-upgrade and post-upgrade operator versions.
Diff the final TOML, ConfigMap key or name, configuration hash, and PodTemplate for each component.
List the workloads that will roll, their Pod counts, and the maximum number of unavailable Pods per group.
Perform an actual upgrade in the development environment and record component start and completion times, rollout overlap, in-place updates versus Pod recreation, readiness, and service availability.
Use the results to define staged rollout plans, maintenance windows, release notes, and rollback procedures.
Acceptance criteria
Attach a before-and-after configuration diff for at least one representative real development tenant.
Document which components will roll and the expected number of Pod restarts.
Verify that concurrent cross-component rollouts do not create Log/DN/TN quorum or service availability risks.
Verify that the fileservice array ordering change does not affect MO behavior.
Document the release sequence, maintenance window, and rollback procedure.
Evaluate whether a pre-upgrade preflight or dry-run configuration diff and an automated regression test should be added.
Background
PR #578 (commit
3de66ab14e3ad0172d686142a58f0a9420ca6e8d) changed parts of the TOML configuration handling from full replacement to deep merge. The follow-up fix in #602 will also preserve typed map fields from user configuration.As a result, any field in an existing cluster configuration that is not overridden by the operator but is preserved by the new merge behavior may appear in the final ConfigMap and change its content hash. Upgrading the operator may therefore trigger rolling updates of the affected MO workloads even when the user has not changed the CR.
Current behavior
Configuration changes are propagated into the PodTemplate through the following mechanisms:
< 1.3: the configuration digest is included in the ConfigMap name, which changes the Pod volume reference.>= 1.3: the configuration digest is included in the configuration file key and written to the PodTemplate annotationmatrixorigin.io/config-suffix.Therefore, a component is rolled whenever its final rendered configuration changes. Potentially affected MO components include:
WebUI, the operator itself, backup, and other components that do not consume this MO configuration should not restart solely because of this hash change.
No globally enforced rollout order across components
The MatrixOneCluster parent controller synchronizes LogSet, DNSet, CNSet, and ProxySet in source-code order, but it does not wait for one component rollout to finish before updating the next component. The child controllers also reconcile independently and concurrently. In particular, after an operator upgrade and restart, several components may begin rolling at the same time.
The rollout must therefore not rely on a global sequence such as Log first, then DN, then CN, and finally Proxy. Their rollout windows may overlap.
Rollout behavior within each workload
maxUnavailableis 1, and the update policy isInPlaceIfPossible.maxUnavailableandmaxSurgeare controlled by the CN configuration. If not explicitly configured,maxUnavailabledefaults to approximately 20%, and there is no stable Pod-name update order.maxUnavailable=1andmaxSurge=0explicitly.maxUnavailabledefaults to approximately 20%, and there is no stable Pod-name update order.Each CN group has its own CloneSet and availability budget, so multiple CN groups may also roll concurrently.
Change in fileservice ordering
After deep merge, operator-managed fileservice entries are placed first and unmatched user entries are appended. The previous behavior usually emitted only the
LOCAL,SHARED, andETLentries.MO currently looks up fileservices primarily by name rather than by array position, but it initializes every additional entry. Correctness risks related to stale fields and extra entries are tracked in #605. This issue focuses on the rollout scope and release safety of the operator upgrade.
Risks
Proposed validation
Acceptance criteria
References