[Bug] Observer/follower keeps forwarding statements to old (degraded) master after master failover, clients receive errors for minutes

[bosswnx]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Doris 3.0.8 (also verified against master branch code)

What's Wrong?

After a master FE failover (old master lost leadership due to OOM/GC pause but the process stayed alive and kept serving its MySQL/thrift ports), a non-master FE (observer/follower) with a lagging journal replay keeps forwarding statements to the old master, and clients continuously receive:

ERROR 1105 (HY000): errCode = 2, detailMessage = The statement has been forwarded to master FE(<old master ip>) and failed to execute because Master FE is not ready. You may need to check FE's status

The error persists until the observer's journal replay catches up (or the observer is restarted), because:

1. The forward target is Env.getMasterHost()/getMasterRpcPort() (MasterOpExecutor → FEOpExecutor), which is backed by the in-memory masterInfo field.
2. masterInfo is only updated by replaying the OP_MASTER_INFO_CHANGE journal entry written by the new master (EditLog.java → Env.setMaster()), or by loading an image at startup. There is no active master re-discovery on the forward path.
3. Meanwhile the old master process is still listening on its ports, so the forwarded thrift call succeeds at the transport level — it reaches StmtExecutor on the old master, which (as a proxy-forwarded statement, isProxy=true) detects it cannot find a valid master and throws the error above (StmtExecutor.executeByNereids/executeByLegacy, the isProxy branch) instead of the sender learning "this node is not master, retry elsewhere".
4. LB health checks (TCP-level) keep routing client connections to the degraded old master as well, compounding the issue.

Additionally, for FORWARD_WITH_SYNC statements (e.g. CREATE USER), the old master's error response still carries a maxJournalId, so the observer blocks in JournalObservable.waitOn() (up to query_timeout * 1.2, default ~18 min) before surfacing anything — the client just hangs.

What You Expected?

After a master failover, the non-master FE should stop forwarding statements to the old (degraded) master quickly — regardless of its journal replay lag. Either:

• the forwarded request should be rejected early by the old master with a NOT_MASTER-style response (so the sender can learn the real master and retry), or
• the sender (FEOpExecutor) should re-discover the current master on forward failure/NOT_MASTER and retry against the real master,

so that statements executed via the observer succeed (or fail fast with a clear, actionable error) instead of failing with "Master FE is not ready" (or hanging in journal-sync wait) for the whole meta_delay_toleration_second window.

How to Reproduce?

Reproduced deterministically on Doris 3.0.8 with 4 FEs in docker (fe1=master, fe2/fe3=followers, fe4=observer). nsenter-based iptables is used to simulate the degraded states (equivalent to what an OOM/GC-pause produces):

1. Freeze the observer's journal replay: block fe4 → fe2/fe3 on the bdbje port (9010), so fe4 cannot replay the new master's journals (its masterInfo stays = fe1).
2. Kill fe1 (docker kill, simulating the OOM death of the old master). fe2 is elected as the new master.
3. Restart fe1 with its bdbje isolated (block 9010 toward fe2/fe3/fe4). fe1 comes back "alive but degraded": MySQL (9030) and thrift (9020) ports still serve, but it is no longer the master and cannot rejoin/catch up — exactly the state of an OOM-recovered old master that lost the election.
4. Execute a forward-to-master statement via the observer (fe4):

$ mysql -h <fe4> -P 9030 -u root -e 'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")'
ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status

(The IP in the message is the old master's own IP — the error is thrown by the old master itself.)

Evidence from the logs:

• Observer (fe4) keeps forwarding to the old (dead/degraded) master after the switch:

  [fe4 log] MasterOpExecutor.forward: forward to master FE TNetworkAddress(hostname:172.20.80.2, port:9020)
  

• Old master (fe1) throws the error on the proxy-forwarded statement:

  [fe1 log] StmtExecutor.executeByLegacy:1042 execute Exception.
  org.apache.doris.common.UserException: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status
  

Notes:

• With a FORWARD_WITH_SYNC statement (CREATE USER), the client instead hangs in journal-sync wait on the observer for up to ~18 minutes before failing.
• The recovery observed in production (rolling restart of the observer) matches this analysis: a restart forces an image load + full journal replay, refreshing masterInfo to the new master.
• meta_delay_toleration_second (default 300s) eventually marks the observer not-ready, which changes the failure mode but does not fix the stale-forward window (5 minutes of broken/hanging statements even in the best case).

Anything Else?

Related prior work:

• [feature](thrift) Add FE thrift rpc redirect master address #25371 added master_address to thrift responses for BE→FE RPCs so the BE can skip an old master, but the FE→FE forward() path never got the equivalent treatment.
• [fix](query) avoid missing result packet for query during master switch #62721 fixed a result-packet loss during master switch on this same forward path, confirming this area is fragile during master transitions.
• BE-side protection against stale masters exists (e.g. commit f13bd7a "follower fe invalidate dns and be rpc skip old master" probes other FEs after a NOT_MASTER/transport failure), while the FE forward path still relies solely on journal-replay-updated masterInfo.

Suggested fix directions (for discussion):

1. In FrontendServiceImpl.forward(): when the receiving FE is not the master, return a NOT_MASTER-style response immediately (with the real master address as a hint, similar to [feature](thrift) Add FE thrift rpc redirect master address #25371) instead of executing the statement deep into StmtExecutor and throwing "Master FE is not ready".
2. In FEOpExecutor.forward(): on receiving NOT_MASTER (or on forward failure), re-discover the current master (e.g. via Env.getHaProtocol().getLeader() which asks bdbje directly and does not depend on journal replay) and retry once against the real master.

This combination would close the stale-forward window regardless of journal replay lag, and also prevents the journal-wait hang caused by the old master's stale maxJournalId in error responses.

Full reproduction scripts and timelines available if needed.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Preliminary assessment

This is a high-confidence FE high-availability bug, primarily affecting availability rather than showing evidence of metadata corruption. The issue is currently open, unassigned, and has no labels. The supplied logs demonstrate the stale forwarding target and the receiver-side proxy error; I did not independently run the four-FE reproduction.

I inspected the nearest published 3.0.8 upstream tag, 3.0.8-rc02 (96abadcc8fd), because upstream does not currently publish an exact 3.0.8 tag, and current master at 3ba29643fa0. The relevant mechanism is present in both.

Code-confirmed facts

• MasterOpExecutor constructs its thrift target from Env.getMasterHost()/getMasterRpcPort(). FEOpExecutor.forward() retries only the same address after a transport exception; it performs no role check or master rediscovery.
• For an already-running non-master FE, the applicable masterInfo refresh path is replay of OP_MASTER_INFO_CHANGE (EditLog -> Env.setMaster()); startup image loading is the other relevant refresh path. There is no active lookup on the SQL forwarding path.
• FrontendServiceImpl.forward() has no master-role gate before proxy execution. On a non-master receiver, the isProxy guard in StmtExecutor produces the reported error instead of a structured redirect.
• ConnectProcessor.proxyExecute() sets the receiver's maxJournalId for both successful and failed execution. For sync forwarding, MasterOpExecutor then waits on that ID without first checking the returned error status. With the default query_timeout=900s and the 1.2 coefficient, the maximum wait is 1080 seconds (18 minutes). This confirms the reported hang mechanism when the observer's replayed ID is below the old FE's returned ID.
• TMasterOpResult currently has neither a structured NOT_MASTER status nor a master-address hint, so the sender cannot distinguish this condition from an ordinary SQL error.

The code evidence supports a failover availability defect. It does not indicate that the old FE executes this forwarded DDL: the proxy guard rejects it before the command runs.

Important constraints on the proposed fix

1. BDBHA.getLeader() is independent of journal replay, but it still calls BDB JE ReplicationGroupAdmin and returns the leader's edit-log/BDB address. Under the reproduction's deliberate observer-to-quorum port-9010 isolation, it may be unable to discover a leader at all. It also does not directly return the FE RPC endpoint. This can be one discovery source, but it cannot by itself guarantee recovery for the stated partition.
2. A hint derived from the rejecting FE's cached masterInfo is not necessarily authoritative in this failure mode; the degraded old master may still point to itself. The sender must reject same-address/looping hints and have a bounded fallback, such as probing known FE RPC endpoints for a node that explicitly confirms it is master.
3. Rediscovery/retry should be scoped to MasterOpExecutor or guarded by an explicit requireMaster request flag. Generic FEOpExecutor is also used intentionally against specific non-master FEs, including all-FE configuration propagation and cross-FE query kill; globally redirecting it would change those semantics.
4. A mutating/non-idempotent statement may be retried safely after an explicit receiver-side, pre-execution NOT_MASTER response. It must not be retried after an ambiguous timeout/transport failure, where execution may already have occurred. Existing shouldNotRetry behavior for non-query operations should be preserved for that case.
5. A structured NOT_MASTER result must bypass JournalObservable.waitOn(); otherwise the protocol addition alone will not fix the sync-forward hang.

Information still useful from the reporter

• The complete FE build version/commit hash for the binary called 3.0.8, plus the full reproduction scripts, so the scenario can become a repeatable integration test.
• For the CREATE USER hang, timestamped lines from observer, old master, and new master showing role transitions, masterInfo, observer replayedJournalId, returned/target maxJournalId, isReady/canRead, and the session query_timeout. The source proves the wait is possible, but those values would verify the exact observed 18-minute timeline.
• The production failover window's BDB JE state-change/replay logs and LB health-check configuration. Please redact credentials and unrelated environment details.

Suggested next steps

1. Triage this as an FE/HA availability bug and reproduce it on the reporter's exact build and current master.
2. Define a backward-compatible master-directed forwarding signal and structured NOT_MASTER response, with an optional hint. Reject master-only requests before proxyExecute() and return no journal-sync target for that rejection.
3. Add bounded redirect/discovery in the master-directed caller only. Prefer a verified hint or BDB leader mapping when available, with known-FE probing as the partition-tolerant fallback; retry mutations only after the explicit pre-execution rejection.
4. Add tests covering an alive old thrift endpoint with stale masterInfo, both FORWARD_NO_SYNC and FORWARD_WITH_SYNC, unavailable BDB discovery, no journal wait on rejection, redirect-loop prevention, and no duplicate mutating execution.
5. Until a fix is available, remove degraded/lagging FEs from the client LB using FE role/readiness/journal progress rather than TCP-port health alone; restarting the stale observer is a recovery action, not a complete fix.

Breakwater-GitHub-Analysis-Slot: slot_ee333f2eee46

[bosswnx]

As requested in the review, here are the full reproduction scripts and the timestamped evidence timeline.

Build version

• Image: official apache/doris:fe-3.0.8 from Docker Hub (manifest digest below). No internal patches.

  $ docker inspect apache/doris:fe-3.0.8 --format '{{index .RepoDigests 0}}'
  apache/doris@sha256:81c72270254585376d8e46239ed90b4f5c04b0d77ae0a0accafb4ea90f7d54c8
  

• Version reported by the FE: doris-0.0.0-c7e635238f4 (from show frontends). Note this is the version string baked into the 3.0.8 release image; the same code path was also confirmed on current master at the source level in the issue body.

Reproduction scripts

docker-compose.yml (4 FE, no BE — only forward-to-master statements are used)

version: '3'
services:
  fe1:  # initial MASTER (simulates the customer's old master 10.206.172.8)
    image: "apache/doris:fe-3.0.8"
    container_name: "repro-fe1"
    hostname: "fe1"
    cap_add: [NET_ADMIN]
    environment:
      - FE_SERVERS=fe1:172.20.80.2:9010,fe2:172.20.80.3:9010,fe3:172.20.80.4:9010,fe4:172.20.80.5:9010
      - FE_ID=1
    ports: ["18030:8030", "19030:9030"]
    volumes:
      - ./fe1/meta:/opt/apache-doris/fe/doris-meta
      - ./fe1/log:/opt/apache-doris/fe/log
    networks:
      doris_net:
        ipv4_address: 172.20.80.2
  # fe2 = 172.20.80.3 FOLLOWER, fe3 = 172.20.80.4 FOLLOWER, fe4 = 172.20.80.5 OBSERVER
  # (identical blocks with FE_ID=2/3/4; fe4 is registered via
  #  ALTER SYSTEM DROP FOLLOWER "172.20.80.5:9010"; ALTER SYSTEM ADD OBSERVER "172.20.80.5:9010"
  #  because the image entrypoint only registers followers)
networks:
  doris_net:
    ipam:
      config:
        - subnet: 172.20.80.0/24

Injection script (repro_demo.sh)

Uses nsenter -t <container-pid> -n iptables from the host — the image itself has no iptables binary.

MYSQL4="mysql -h 127.0.0.1 -P 19033 -u root"   # observer fe4 (customer's LB -> 10.206.172.49)
pid() { docker inspect repro-$1 --format '{{.State.Pid}}'; }

# STEP 1: freeze observer's journal replay toward the potential new masters (bdbje 9010)
FE4PID=$(pid fe4)
for ip in 172.20.80.3 172.20.80.4; do
    nsenter -t $FE4PID -n iptables -A INPUT  -p tcp -s $ip --dport 9010 -j DROP
    nsenter -t $FE4PID -n iptables -A OUTPUT -p tcp -d $ip --dport 9010 -j DROP
done

# STEP 2: old master "OOM" death
docker kill repro-fe1

# STEP 3: wait for the new master election (fe3 won in this run)

# STEP 4: restart fe1, wait for its thrift (9020) to listen, then isolate its bdbje:
#         alive-but-degraded old master (mysql/thrift serving, cannot rejoin)
docker start repro-fe1
# ... wait for /dev/tcp/172.20.80.2/9020 ...
FE1PID=$(pid fe1)
for ip in 172.20.80.3 172.20.80.4 172.20.80.5; do
    nsenter -t $FE1PID -n iptables -A INPUT  -p tcp -s $ip --dport 9010 -j DROP
    nsenter -t $FE1PID -n iptables -A OUTPUT -p tcp -d $ip --dport 9010 -j DROP
done

# STEP 5: customer scenario — statement via the observer
mysql -h 127.0.0.1 -P 19033 -u root -e \
  'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")'

Client-visible result (FORWARD_NO_SYNC statement)

$ mysql -h 127.0.0.1 -P 19033 -u root -e 'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")'
ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status

fe4 audit log confirms this exact error was returned to the client:

AuditLogger 2026-08-28 11:43:18,657 [query] | ... |Timestamp=2026-08-28 11:43:18.657|Client=172.20.80.1:...|FeIp=172.20.80.5|State=ERR|ErrorCode=1105|ErrorMessage=errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status|...|Stmt=ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")

Timestamped evidence timeline (all times are container-log times, same host clock)

Time	Node	Event (log line)
11:37:57	fe4 (observer)	setMaster MasterInfo: host=172.20.80.2 (masterInfo = fe1, normal)
11:38:33	fe4	replayed journal id is 48, replay to journal id is 49 (replay healthy before injection)
11:38:37	fe1	docker kill — old master dies
11:38:37	fe4	transfer FE type from OBSERVER to UNKNOWN (lost its feeder)
11:39:00–11:39:10	fe3	transfer FE type from UNKNOWN to MASTER — fe3 (172.20.80.4) is the new master
11:41:41	fe1	restarted; transfer FE type from INIT to FOLLOWER (thrift 9020 up; bdbje then isolated by iptables)
11:41:42	fe1	setMaster MasterInfo: host=172.20.80.4 — briefly learned the new master from residual replay, then isolated
11:42:12	fe4	forward to master FE TNetworkAddress(hostname:172.20.80.2, port:9020), statement id: 12 — observer still forwards to the old master fe1 (its masterInfo was never refreshed: replay frozen since 11:38:37)
11:42:13.114	fe1	StmtExecutor.executeByLegacy():1042 execute Exception → UserException: ... The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready (thrown by the old master itself, stack: proxyExecute → FrontendServiceImpl.forward)
11:42:13.117	fe4	forwarding to master get result max journal id: 79 → JournalObservable.waitOn(): waiting for the observer to replay journal to 79 with timeout: 1080000 ms — the sync-wait hang begins even though the result carried an error
11:43:18	fe4	(with the 25 s client timeout override in this test run) error surfaces to the client; with the default query_timeout=900, this wait would last up to 1080 s

Notes on the numbers requested in the review:

• Observer replayedJournalId was frozen at ~49 since 11:38:37; the old master returned maxJournalId=79 (its own last replayed id), so waitOn(79) can never be satisfied while the replay freeze lasts — this is the exact WITH_SYNC hang mechanism.
• meta out of date ... delta:300001ms, hasLog:false, feType:UNKNOWN appears at ~300 s after the freeze (default meta_delay_toleration_second=300), after which the observer's failure mode changes to Node catalog is not ready — consistent with the review's note that the stale-forward window is bounded by but not fixed by this config.
• In the 11:42 attempt above, fe1's own masterInfo had already flipped to fe3 (via its brief residual replay at 11:41:42 before isolation). The customer's real-world case is harsher: an OOM-recovered old master that never replays the new master's journals keeps masterInfo = itself, which is why the hint-from-rejecting-FE approach must not be trusted blindly (matches review constraint 2).

FORWARD_WITH_SYNC variant (CREATE USER)

Same injection; CREATE USER fin_1 IDENTIFIED BY 'p' sent via fe4 while fe1 is dead-but-forwardable produced either:

• ForwardToMasterException ... cause: EOF, Socket is closed by peer (fe1 down), or
• when fe1 is in the degraded-alive state: the statement is rejected by fe1 with the same "Master FE is not ready" error, but fe4 blocks in JournalObservable.waitOn(79) for the full timeout (only interrupted by client-side SET query_timeout — with query_timeout=5 the wait log shows timeout: 6000 ms).

This confirms the review's constraint 5: a structured NOT_MASTER response alone would not fix the sync-forward hang; the rejection path must bypass JournalObservable.waitOn().

What I don't have

• The customer's production LB health-check config and BDB JE state-change logs from the real failover window (their environment; I'll request them). The reproduction above is my own environment.

[zhang-arvin]

I will work on this.