diff --git a/docs/05-administrator-guide/03-operations/03-node-decommissioning-and-maintenance/03-datanodes/01-datanode-decommission.md b/docs/05-administrator-guide/03-operations/03-node-decommissioning-and-maintenance/03-datanodes/01-datanode-decommission.md index 7f0cf1c6ae..377a2c4c73 100644 --- a/docs/05-administrator-guide/03-operations/03-node-decommissioning-and-maintenance/03-datanodes/01-datanode-decommission.md +++ b/docs/05-administrator-guide/03-operations/03-node-decommissioning-and-maintenance/03-datanodes/01-datanode-decommission.md @@ -121,3 +121,7 @@ Upon restart, SCM enters Safemode and rebuilds its cluster membership list from :::note There is currently no CLI command to manually "forget" a node without a restart. ::: + +## See also + +- [Troubleshooting Decommissioning and Maintenance](../../../../troubleshooting/decommission-maintenance) for a decommission that is slow or stuck diff --git a/docs/06-troubleshooting/10-decommission-maintenance.md b/docs/06-troubleshooting/10-decommission-maintenance.md index 37f0f00a31..4a635e3ce3 100644 --- a/docs/06-troubleshooting/10-decommission-maintenance.md +++ b/docs/06-troubleshooting/10-decommission-maintenance.md @@ -1,8 +1,131 @@ --- -draft: true sidebar_label: Decommissioning and Maintenance --- # Troubleshooting Decommissioning and Maintenance -**TODO:** File a subtask under [HDDS-9860](https://issues.apache.org/jira/browse/HDDS-9860) and complete this page or section. +A Datanode moves from `DECOMMISSIONING` to `DECOMMISSIONED` only after every container it holds is sufficiently replicated on other nodes. +On a dense Datanode this takes hours, and on a large or busy cluster it can take days. +Slow is normal; stuck is not. +This page shows how to tell the two apart and what to do in each case. + +SCM re-checks every decommissioning node each `ozone.scm.datanode.admin.monitor.interval` (default `30s`), and completes it once: + +1. All pipelines on the node, Ratis and EC, have closed. +2. The node is alive and has reported its new operational state in a heartbeat. +3. Every container on the node is `CLOSED` or `QUASI_CLOSED` and has enough healthy replicas on other nodes. Containers in `DELETING` or `DELETED` state are ignored. + +:::note Maintenance +A node entering maintenance goes through the same checks with a lower replica threshold: a Ratis container needs `hdds.scm.replication.maintenance.replica.minimum` (default `2`) replicas on other nodes, and an EC container needs its data replicas plus `hdds.scm.replication.maintenance.remaining.redundancy` (default `1`). +The status command below only lists `DECOMMISSIONING` nodes, so for a node stuck in `ENTERING_MAINTENANCE` use the [SCM log](#scm-log) and [metrics](#metrics) instead. +See [Datanode Maintenance Mode](../administrator-guide/operations/node-decommissioning-and-maintenance/datanodes/datanode-maintenance) for details. +::: + +## Check the progress + +### Decommission status command + +Run the status command a few times, some minutes apart: + +```shell +ozone admin datanode status decommission [--node-id= | --ip=] [--json] +``` + +```text +Decommission Status: DECOMMISSIONING - 1 node(s) + +Datanode: 6f1c...-...-... (/default-rack/10.0.0.12/dn12.example.com) +Decommission Started At : 01/09/2026 02:14:05 UTC +No. of Unclosed Pipelines: 0 +No. of UnderReplicated Containers: 1834 +No. of Unclosed Containers: 3 +{UnderReplicated=[#1001, #1002, ...], UnClosed=[#1200, #1201, #1202]} +``` + +| What you see | Next step | +| --- | --- | +| `No. of UnderReplicated Containers` keeps decreasing | The decommission is progressing. If it is too slow, see [Make it faster](#make-it-faster). | +| `No. of Unclosed Pipelines` stays above zero | See [Pipelines do not close](#pipelines-do-not-close). | +| `No. of Unclosed Containers` stays above zero | See [Containers do not close](#containers-do-not-close). | +| `No. of UnderReplicated Containers` stays flat for several monitor intervals | See [Replication is stalled](#replication-is-stalled). | + +### SCM log + +On every monitor run SCM logs one line per node. +Compare consecutive lines to see whether `underReplicated` moves: + +```text + has 12405 sufficientlyReplicated, 17 deleting, 1834 underReplicated and 3 unclosed containers +``` + +While pipelines are still open, the line is `Waiting for pipelines to close for . There are N pipelines` instead. +Each blocking container is logged as `Under Replicated Container ...` or `Unclosed Container ...`, with the state and location of its replicas, up to `ozone.scm.datanode.admin.monitor.logging.limit` (default `1000`) lines per node and category. + +### Metrics + +Import the [Ozone Datanode Decommission and Maintenance dashboard](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone%20-%20Datanode%20Decommission%20and%20Maintenance.json) into [Grafana](../administrator-guide/operations/observability/grafana). +It graphs the same counters as the status command, per node and in total, next to the SCM Replication Manager and Datanode replication metrics used on the rest of this page. +In a healthy decommission, under-replicated containers trend down and sufficiently replicated containers trend up. +Each metric is described in the [Datanode decommission guide](../administrator-guide/operations/node-decommissioning-and-maintenance/datanodes/datanode-decommission#metrics). + +## Pipelines do not close + +A small non-zero count during the first few minutes is expected: a closed pipeline is only removed from the node by the pipeline scrubber (`ozone.scm.pipeline.scrub.interval`, default `5m`) after it has stayed `CLOSED` for `ozone.scm.pipeline.destroy.timeout` (default `66s`). +If the count stays above zero for longer: + +1. Run `ozone admin pipeline list` and find the pipelines that still include the node. +2. Check that the other Datanodes in those pipelines are healthy. + +## Containers do not close + +A container blocks the workflow while it is `OPEN` or `CLOSING`, and should close within a few heartbeats after its pipeline closes. +If it does not: + +1. Take the container IDs from the `UnClosed` list of the status command, or from the `Unclosed Container` lines in the SCM log. +2. Run `ozone admin container info ` to see its state and replicas. +3. Check the Datanode logs on the replica nodes for errors closing that container. + +A `QUASI_CLOSED` container does not block the workflow by itself, as long as its healthy replicas on `IN_SERVICE` nodes are also `QUASI_CLOSED`. + +## Replication is stalled + +Take a few IDs from the `UnderReplicated` list, search the SCM log for them, and match the message: + +- **`Cannot replicate container because no suitable targets were found`**: no Datanode can take a new replica. + A target must be `IN_SERVICE`, healthy, have enough free space, not already hold a replica, and satisfy the rack placement policy. + Add capacity, or recommission some nodes with `ozone admin datanode recommission` and decommission them in smaller batches. +- **`No sources with capacity available for replication of container `**: every node that could serve as source is at its replication limit, and `ReplicateContainerCmdsDeferredTotal` climbs. + Raise the per-Datanode limits, see [Make it faster](#make-it-faster). +- **`Datanode is dead and the admin workflow cannot continue`**: the decommissioning node died, and SCM put it back to `IN_SERVICE`. + Its containers are re-replicated as ordinary under-replicated containers; if the node comes back, decommission it again. + +If there is no such message but `ReplicationCmdsSentTotal` grows much faster than `ReplicasCreatedTotal`, the copies are failing or timing out. +Check `numFailureReplications` and `numTimeoutReplications` in the Datanode `ReplicationSupervisorMetrics`. +Each copy must finish within `hdds.scm.replication.event.timeout` (default `12m`), otherwise SCM retries it; for large containers on a slow link, raise the timeout or lower the replication limits so that each copy gets more bandwidth. + +## Make it faster + +Replication is throttled per Datanode and cluster-wide, and those throttles set the pace of the decommission. +Raise the one that is limiting you, in steps, and watch client latency, since every step adds replication load to the cluster. + +| Symptom | Property to raise | +| --- | --- | +| `ReplicateContainerCmdsDeferredTotal` climbs | `hdds.scm.replication.datanode.replication.limit`, or `hdds.datanode.replication.outofservice.limit.factor` to raise it only for decommissioning nodes | +| SCM logs `The maximum number of pending replicas () are scheduled`, and `PendingReplicationLimitReachedTotal` climbs | `hdds.scm.replication.inflight.limit.factor` | +| `queueTime` of the `MeasuredReplicator` metrics is high on the decommissioning node | `hdds.datanode.replication.streams.limit`, or `hdds.datanode.replication.per.volume.streams.limit` if per-volume replication is enabled | + +| Property | Default | Set on | Effect | +| --- | --- | --- | --- | +| `hdds.scm.replication.datanode.replication.limit` | `20` | SCM | Replication load that can be queued on one Datanode, counted as replication commands plus EC reconstruction commands × `hdds.scm.replication.datanode.reconstruction.weight` (default `3`). Reconfigurable. | +| `hdds.datanode.replication.outofservice.limit.factor` | `2.0` (`1`-`10`) | SCM **and** Datanodes | Multiplies the limit above, and the Datanode replication queue and thread pools, on decommissioning and maintenance nodes. Needs a restart. | +| `hdds.scm.replication.inflight.limit.factor` | `0.75` | SCM | Scales the cluster-wide cap of *healthy nodes × replication limit* replicas pending creation. `0` disables the cap. Reconfigurable. | +| `hdds.datanode.replication.streams.limit` | `10` | Datanodes | Replication threads on a Datanode. Limits outgoing copies only while `hdds.datanode.replication.per.volume.enabled` is `false` (default). Reconfigurable. | +| `hdds.datanode.replication.per.volume.streams.limit` | `2` | Datanodes | Outgoing copies per data volume when `hdds.datanode.replication.per.volume.enabled` is `true`. Reconfigurable. | + +With the defaults, a decommissioning node can have 20 × 2 = 40 replication commands queued (each EC reconstruction command counts as 3), and all decommissioning nodes share a cluster-wide cap of *healthy nodes × 20 × 0.75* replicas pending creation. +Reconfigurable properties can be changed without a restart, see [Dynamic Property Reload](../administrator-guide/operations/dynamic-property-reload). + +:::note +Raising the limits only helps if replication is slowly progressing. +If the under-replicated count is flat, go back to [Replication is stalled](#replication-is-stalled) first. +:::