From 51140105eb522fbe9081700088c4c522086f2733 Mon Sep 17 00:00:00 2001 From: Oladele Oluwaseun Date: Thu, 30 Jul 2026 23:18:06 +0100 Subject: [PATCH 1/3] docs-clarify-throttle-control-replicas --- doc/throttle.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/throttle.md b/doc/throttle.md index 923d6f64e..4ef760133 100644 --- a/doc/throttle.md +++ b/doc/throttle.md @@ -30,6 +30,15 @@ Otherwise you may specify your own list of replica servers you wish it to observ Note that you may dynamically change both `--max-lag-millis` and the `throttle-control-replicas` list via [interactive commands](interactive-commands.md) + +### Important: choosing the right hosts for `--throttle-control-replicas` + +The hosts you list here should be genuine replicas within the actual replication topology of the table being migrated — not arbitrary hosts, and not replicas of an unrelated database or instance. + +It's possible to point `--throttle-control-replicas` at hosts that gh-ost can successfully connect to and poll for lag, even if those hosts have no replication relationship to the table actually being migrated. In that scenario, gh-ost will report healthy, low lag numbers throughout the migration — but this gives a false sense of safety, because the replica that actually matters (e.g., one serving downstream read traffic or analytics for the table being changed) is never being monitored at all, and could be silently falling behind without triggering any throttling. + +Before running a migration, confirm the hosts you're passing to `--throttle-control-replicas` are actually replicating from the same primary and the same table you're migrating — ideally, the specific replica(s) that downstream consumers of that table's data depend on. + #### Status thresholds - `--max-load`: list of metrics and threshold values; topping the threshold of any will cause throttler to kick in. From ba308c06f8fbcfc969074377c924da46b16f620c Mon Sep 17 00:00:00 2001 From: Oladele Oluwaseun Date: Fri, 31 Jul 2026 10:43:54 +0100 Subject: [PATCH 2/3] docs: condense throttle-control-replicas note per review feedback --- doc/throttle.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/doc/throttle.md b/doc/throttle.md index 4ef760133..b221575b9 100644 --- a/doc/throttle.md +++ b/doc/throttle.md @@ -24,6 +24,8 @@ Otherwise you may specify your own list of replica servers you wish it to observ - `--throttle-control-replicas`: list of replicas you explicitly wish `gh-ost` to check for replication lag. +- _Note:_ these hosts should be actual replicas within the migrated table's replication chain — pointing this at unrelated hosts can report healthy lag while leaving your real downstream replicas unmonitored. + Example: `--throttle-control-replicas=myhost1.com:3306,myhost2.com,myhost3.com:3307` - `--max-lag-millis`: maximum allowed lag; any controlled replica lagging more than this value will cause throttling to kick in. When all control replicas have smaller lag than indicated, operation resumes. @@ -31,14 +33,6 @@ Otherwise you may specify your own list of replica servers you wish it to observ Note that you may dynamically change both `--max-lag-millis` and the `throttle-control-replicas` list via [interactive commands](interactive-commands.md) -### Important: choosing the right hosts for `--throttle-control-replicas` - -The hosts you list here should be genuine replicas within the actual replication topology of the table being migrated — not arbitrary hosts, and not replicas of an unrelated database or instance. - -It's possible to point `--throttle-control-replicas` at hosts that gh-ost can successfully connect to and poll for lag, even if those hosts have no replication relationship to the table actually being migrated. In that scenario, gh-ost will report healthy, low lag numbers throughout the migration — but this gives a false sense of safety, because the replica that actually matters (e.g., one serving downstream read traffic or analytics for the table being changed) is never being monitored at all, and could be silently falling behind without triggering any throttling. - -Before running a migration, confirm the hosts you're passing to `--throttle-control-replicas` are actually replicating from the same primary and the same table you're migrating — ideally, the specific replica(s) that downstream consumers of that table's data depend on. - #### Status thresholds - `--max-load`: list of metrics and threshold values; topping the threshold of any will cause throttler to kick in. From 8177de02568d63ccfc8d8fd1c52a52e8a94c3aac Mon Sep 17 00:00:00 2001 From: Oladele Oluwaseun Date: Tue, 15 Sep 2026 15:04:13 +0100 Subject: [PATCH 3/3] Create postpone-cut-over.md --- doc/postpone-cut-over.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/postpone-cut-over.md diff --git a/doc/postpone-cut-over.md b/doc/postpone-cut-over.md new file mode 100644 index 000000000..d534a344d --- /dev/null +++ b/doc/postpone-cut-over.md @@ -0,0 +1,26 @@ + +### Postponing Cut-over vs. Throttling for Maintenance Windows + +When managing long-running migrations (e.g. 100M+ rows / multi-hour executions), it is important to distinguish between **throttling** and **postponing cut-over**: + +| Operational Feature | Throttling (`--throttle-additional-flag-file` / `echo throttle`) | Postponing Cut-over (`--postpone-cut-over-flag-file` / `echo postpone`) | +| :--- | :--- | :--- | +| **Row Copying** | Paused | Completes to 100% | +| **Streamer & Changelog** | Suspended / idle | **Active** (tails binlogs in real-time, near 0s lag) | +| **Heartbeat Injector** | Suspended (HeartbeatLag grows) | **Active** (keeps connection alive) | +| **MySQL `wait_timeout` Risk** | High (idle connections can disconnect if throttled >8h) | **None** (continuous stream prevents connection timeouts) | +| **Database Load** | 0% | Near 0% (only processes incoming live DML events) | +| **Recommended Use Case** | Immediate load shedding during unexpected DB spikes | **Scheduling cut-over table swap for off-peak maintenance windows** | + +#### Recommended Multi-Hour Workflow: +1. Start migration with cut-over postponed (`echo postpone`). +2. Allow `gh-ost` to complete row copying in the background while keeping replication lag minimal. +3. During your designated maintenance window, trigger cut-over with `echo unpostpone` (or remove the postpone flag file) for an instant (<1-2s) table swap. + +### Note on UUID / Alphanumeric Primary Keys and Progress Metrics + +When migrating tables keyed by alphanumeric or UUID strings (`VARCHAR(36)`), `gh-ost` iterates across the lexicographical key space (`0000...` to `ffff...`). + +Because initial total row counts rely on MySQL `EXPLAIN` statistics, the estimated total rows may diverge from the actual density of keys traversed. During such migrations: +- Progress may display percentages greater than 100% (e.g. `139%`, `164%`). +- `ETA: due` indicates that the row copy has traversed past the initial estimate and is in the final portion of the keyspace. Operators should inspect the highest key copied in the ghost table relative to `MAX(pk)` on the original table to observe real-time boundary progress.