Bug Type (问题类型)
server status (启动/运行异常)
Before submit
Environment (环境信息)
- Server Version:
hugegraph/server:latest published image, built 2026-09-08, hg-store-client-1.7.0.jar. Also seen on images built from master 60c8803d and from 6ec19838.
- Backend: HStore, 3 PD + 3 Store + 3 Server, replica count 3
- OS: 12 CPUs, 15 G RAM, Ubuntu, Kubernetes v1.37.0 (kind), Docker 29.5.3
- Data Size: none, this happens on a completely empty first install
Expected & Actual behavior (期望与实际表现)
Expected. On a cold start of a distributed cluster, a Server that finds the Stores have
not yet registered with PD waits for them and then starts.
Actual. The Server asks PD for partition information, gets
error code = 105, The number of active stores is less then 3, retries 10 times over 38
seconds, hits a hard retry ceiling, and the JVM exits 1. Under Kubernetes the container
is restarted and usually succeeds on the second attempt, so the cluster ends up healthy and
the only lasting trace is RESTARTS: 1. Outside an orchestrator that restarts it, the Server
stays down.
This is a startup race, not a timeout. It is not related to the entrypoint startup budget
from #3186: the log contains no The operation timed out(...) line, and the value was 450 s
here.
Case B is the reproduction detailed below. Case A stands for the successful path, which is
what the other two Servers on that same install did; its T+64 s is illustrative, while the
fixed timings (T+42, T+80, T+85) are measured.
Timeline from one reproduction
| Time |
Event |
| 09:44:28 |
Server containers start |
| 09:44:38 to 09:44:44 |
PD pods start, 10 to 16 s after the Server |
| 09:44:39 to 09:44:45 |
Store pods start |
| 09:45:11 |
Server JVM finishes booting (~42 s) and requests partition information |
| 09:45:11 to 09:45:49 |
error code = 105 x10, backoff 1,1,1,2,3,4,5,6,7,8 s = 38 s |
| 09:45:49 |
NodeTxExecutor - the number of retries reached the upper limit : 10 |
| 09:45:54 |
container exits 1, total lifetime 85 s |
Log and stack trace
2026-09-10 09:45:11 [main] [ERROR] o.a.h.s.c.HgStoreNodePartitionerImpl - An error occurred
while getting partition information :PD request error, error code = 105,
msg = The number of active stores is less then 3
2026-09-10 09:45:11 [main] [INFO] o.a.h.s.c.NodeTxExecutor - Waiting 1 seconds for the next try.
... 10 attempts, backoff growing to 8 s ...
2026-09-10 09:45:49 [main] [ERROR] o.a.h.s.c.NodeTxExecutor - the number of retries reached
the upper limit : 10,caused by:
java.lang.RuntimeException: PD request error, error code = 105,
msg = The number of active stores is less then 3
at org.apache.hugegraph.store.client.HgStoreNodePartitionerImpl.partition(HgStoreNodePartitionerImpl.java:87) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxSessionProxy.doPartition(NodeTxSessionProxy.java:865) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxSessionProxy.toNodeTkvList(NodeTxSessionProxy.java:778) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxSessionProxy.getNodeStream(NodeTxSessionProxy.java:921) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxSessionProxy.lambda$createTable$20(NodeTxSessionProxy.java:321) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxExecutor.lambda$isAllTrue$12(NodeTxExecutor.java:344) ~[hg-store-client-1.7.0.jar:1.7.0]
at org.apache.hugegraph.store.client.NodeTxExecutor.lambda$retryingInvoke$15(NodeTxExecutor.java:381) ~[hg-store-client-1.7.0.jar:1.7.0]
The restarted container hits the same error 9 more times and then succeeds, because by
then the Stores have registered.
Why it is intermittent
It is a race between how long the Stores take to register with PD and a fixed 38 second
budget that begins about 42 seconds after the Server container starts. Rates observed across
four campaigns on two different hosts:
| Run |
Host |
Fresh Server starts |
Failed |
| 2026-09-05 |
A |
3 |
1 |
| 2026-09-10 run 1 |
B |
3 |
0 |
| 2026-09-10 run 2 |
B |
3 |
1 |
| 2026-09-10 F1 run |
B |
3 |
1 |
| total |
|
12 |
3 |
So 3 of 12 fresh Server starts failed, and 3 of the 4 installs produced at least one
crashed Server. The per-install figure is the one an operator meets.
Earlier campaigns saw a similar symptom that is deliberately not counted above. On
2026-08-29, two of six Server starts exited 1 on install, but each ran about 2 min 45 s
before dying, which fits the 120 s entrypoint self-kill of #3186 rather than this 38 s retry
ceiling, and the logs were lost so neither can be attributed. #3187 has since fixed that
one. If those two turn out to be this bug as well the rate is higher, not lower.
(Corrected after first posting. The first two rows originally read 6/2 and 10/0. The 6/2
was carried over from #3186, which is the entrypoint's 120 s self-kill and a different
failure, and the 10 was a pod count rather than a count of Server starts. Every campaign
deployed exactly 3 Servers.)
It does not reproduce on pod replacement. 66 Server replacements against an already
running cluster produced zero failures, because a replacement Server asks PD and gets a
straight answer in about 16 seconds. Only a cold cluster has the window. Anyone trying to
reproduce this must recreate the whole cluster, not restart a Server.
Steps to reproduce
- Deploy 3 PD + 3 Store + 3 Server on Kubernetes, all at once, with no ordering between them.
- Watch the Server pods on the very first install.
- Repeat the full install a few times. Three of the four installs observed so far showed
at least one Server with RESTARTS: 1 and lastState.terminated.exitCode: 1.
A note on diagnosing this
The message an operator actually sees is only:
Connecting to HugeGraphServer (http://0.0.0.0:8080/graphs)...........Starting HugeGraphServer failed
See /hugegraph-server/logs/hugegraph-server.log for HugeGraphServer log output.
That file is inside the container and the container is gone, so kubectl logs --previous
returns the line above and nothing else. The stack trace in this report was only obtainable
by mounting a volume at /hugegraph-server/logs before the first boot, so the crashed run's
log survived the restart. Logging the fatal startup cause to stdout would make this class
of failure diagnosable without that trick.
Suggested direction
These are suggestions rather than a diagnosis of the right fix:
- The retry ceiling is
HgStoreClientConst.NODE_MAX_RETRYING_TIMES = 10
(hugegraph-store/hg-store-client/.../util/HgStoreClientConst.java:48), used only by
NodeTxExecutor at lines 376 and 383. Nothing reads it from configuration or the
environment, so 38 seconds is the whole budget an operator gets. A cold cluster bootstrap
can reasonably take longer than that. Either make the budget configurable, or use a longer
one on the bootstrap path where "stores not registered yet" is an expected transient
rather than a fault.
- Treat
error code = 105 during startup as "not ready yet" rather than as a retryable
failure with a short ceiling, since it is precisely the condition that resolves on its own.
- Log the terminal cause to stdout, per the note above.
Orchestration can also side-step this by not starting the Server until PD reports the
expected number of active stores. I will carry that as a gate in the Helm chart in #3131 /
#3132, but it is a workaround for the retry ceiling, not a fix for it.
Related
Vertex/Edge example (问题点 / 边数据举例)
Not applicable, the cluster is empty.
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
Not applicable, the failure happens before any schema exists.
Bug Type (问题类型)
server status (启动/运行异常)
Before submit
Environment (环境信息)
hugegraph/server:latestpublished image, built 2026-09-08,hg-store-client-1.7.0.jar. Also seen on images built from master60c8803dand from6ec19838.Expected & Actual behavior (期望与实际表现)
Expected. On a cold start of a distributed cluster, a Server that finds the Stores have
not yet registered with PD waits for them and then starts.
Actual. The Server asks PD for partition information, gets
error code = 105, The number of active stores is less then 3, retries 10 times over 38seconds, hits a hard retry ceiling, and the JVM exits 1. Under Kubernetes the container
is restarted and usually succeeds on the second attempt, so the cluster ends up healthy and
the only lasting trace is
RESTARTS: 1. Outside an orchestrator that restarts it, the Serverstays down.
This is a startup race, not a timeout. It is not related to the entrypoint startup budget
from #3186: the log contains no
The operation timed out(...)line, and the value was 450 shere.
Case B is the reproduction detailed below. Case A stands for the successful path, which is
what the other two Servers on that same install did; its T+64 s is illustrative, while the
fixed timings (T+42, T+80, T+85) are measured.
Timeline from one reproduction
error code = 105x10, backoff 1,1,1,2,3,4,5,6,7,8 s = 38 sNodeTxExecutor - the number of retries reached the upper limit : 10Log and stack trace
The restarted container hits the same error 9 more times and then succeeds, because by
then the Stores have registered.
Why it is intermittent
It is a race between how long the Stores take to register with PD and a fixed 38 second
budget that begins about 42 seconds after the Server container starts. Rates observed across
four campaigns on two different hosts:
So 3 of 12 fresh Server starts failed, and 3 of the 4 installs produced at least one
crashed Server. The per-install figure is the one an operator meets.
Earlier campaigns saw a similar symptom that is deliberately not counted above. On
2026-08-29, two of six Server starts exited 1 on install, but each ran about 2 min 45 s
before dying, which fits the 120 s entrypoint self-kill of #3186 rather than this 38 s retry
ceiling, and the logs were lost so neither can be attributed. #3187 has since fixed that
one. If those two turn out to be this bug as well the rate is higher, not lower.
(Corrected after first posting. The first two rows originally read 6/2 and 10/0. The 6/2
was carried over from #3186, which is the entrypoint's 120 s self-kill and a different
failure, and the 10 was a pod count rather than a count of Server starts. Every campaign
deployed exactly 3 Servers.)
It does not reproduce on pod replacement. 66 Server replacements against an already
running cluster produced zero failures, because a replacement Server asks PD and gets a
straight answer in about 16 seconds. Only a cold cluster has the window. Anyone trying to
reproduce this must recreate the whole cluster, not restart a Server.
Steps to reproduce
at least one Server with
RESTARTS: 1andlastState.terminated.exitCode: 1.A note on diagnosing this
The message an operator actually sees is only:
That file is inside the container and the container is gone, so
kubectl logs --previousreturns the line above and nothing else. The stack trace in this report was only obtainable
by mounting a volume at
/hugegraph-server/logsbefore the first boot, so the crashed run'slog survived the restart. Logging the fatal startup cause to stdout would make this class
of failure diagnosable without that trick.
Suggested direction
These are suggestions rather than a diagnosis of the right fix:
HgStoreClientConst.NODE_MAX_RETRYING_TIMES = 10(
hugegraph-store/hg-store-client/.../util/HgStoreClientConst.java:48), used only byNodeTxExecutorat lines 376 and 383. Nothing reads it from configuration or theenvironment, so 38 seconds is the whole budget an operator gets. A cold cluster bootstrap
can reasonably take longer than that. Either make the budget configurable, or use a longer
one on the bootstrap path where "stores not registered yet" is an expected transient
rather than a fault.
error code = 105during startup as "not ready yet" rather than as a retryablefailure with a short ceiling, since it is precisely the condition that resolves on its own.
Orchestration can also side-step this by not starting the Server until PD reports the
expected number of active stores. I will carry that as a gate in the Helm chart in #3131 /
#3132, but it is a workaround for the retry ceiling, not a fix for it.
Related
HgStoreNodePartitionerImpl,NodeTxExecutor) but adifferent trigger: stale DNS after Store pod replacement, not a cold-start race.
Vertex/Edge example (问题点 / 边数据举例)
Not applicable, the cluster is empty.
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
Not applicable, the failure happens before any schema exists.