The validator plugin handles block signing and production. It runs a dedicated 250 ms timer loop on its own OS thread, executes a series of safety checks on every tick, and calls database::generate_block() when all conditions are met.
Source: plugins/validator/validator.cpp
chain::plugin, p2p::p2p_plugin, snapshot::snapshot_plugin
| Option | Default | Description |
|---|---|---|
validator / -w |
— | Validator account name(s); may be repeated |
private-key |
— | WIF private key(s) for signing; may be repeated |
emergency-private-key |
— | WIF key for emergency consensus; auto-adds CHAIN_EMERGENCY_VALIDATOR_ACCOUNT to the validator set |
enable-stale-production |
false |
Bypass participation and sync checks (testnet / network recovery only) |
disable-minority-fork-detection |
false |
Skip minority-fork detection entirely (single-operator testnet/fork only). Never auto-cleared by healthy participation — see Minority Fork Detection |
required-participation |
3300 |
Minimum validator participation in basis points (3300 = 33%) |
fork-collision-timeout-blocks |
21 |
Consecutive fork-collision deferrals before forcing production (one full validator round) |
| Option | Default | Description |
|---|---|---|
ntp-server |
pool.ntp.org, time.google.com, time.cloudflare.com |
NTP servers; may be repeated |
ntp-request-interval |
900 |
Normal sync interval in seconds |
ntp-retry-interval |
300 |
Retry interval when no NTP reply |
ntp-round-trip-threshold |
150 |
Discard NTP replies with round-trip > N ms |
ntp-history-size |
5 |
Moving-average window for NTP delta smoothing |
| Option | Default | Description |
|---|---|---|
debug-block-production |
false |
Enable verbose debug logging in the chain database |
The production loop runs on a dedicated production_io_service_ and its own OS thread — completely separate from the AppBase/P2P shared io_service. This prevents P2P activity (peer disconnects, TLS handshakes, send-queue drains) from delaying the 250 ms timer callback.
Tick alignment:
timer tick every 250 ms aligned to wall-clock 250 ms boundaries
minimum sleep: 50 ms (to absorb OS jitter)
Look-ahead: now = ntp_time + 250 ms — shifts the production decision forward so the tick at T_slot - 250 ms aligns exactly to the slot boundary:
Slot at T=6.000s:
Tick at T=5.750 → now=6.000 → slot matched → produce at lag=0ms
Tick at T=6.000 → now=6.250 → lag=250ms → still within 500ms threshold
Lag skip: After a lag result, the same slot would re-trigger every tick for the rest of the slot interval (3 s before HF16, 1 s after). A guard skips ahead to the next slot boundary so the loop yields CPU instead of spinning.
The following checks run in order on every tick where slot > 0.
| # | Check | Failure result |
|---|---|---|
| 1 | DLT sync gate (DLT mode only): chain().is_syncing() is false, or this node is the emergency master |
not_synced |
| 2 | Snapshot pause gate: snapshot().is_snapshot_in_progress() is false |
not_synced |
| 3 | P2P catchup gate: p2p().is_catching_up_after_pause() is false |
not_synced |
| 4 | HF12 three-state safety (see below) | not_synced / low_participation |
| 5 | slot = db.get_slot_at_time(now) > 0 |
not_time_yet |
| 6 | Scheduled validator is in our configured set | not_my_turn |
| 7 | Slot not already filled (scheduled_time > head_block_time) |
not_time_yet |
| 8 | Validator's on-chain signing_key is non-zero |
not_my_turn |
| 9 | Private key for signing_key is loaded |
no_private_key |
| 10 | Pre-HF12: participation ≥ threshold | low_participation |
| 11 | ` | scheduled_time - now |
| 12 | Fork collision check (see below) | fork_collision |
| 13 | Second snapshot pause check (race window) | not_time_yet |
| 14 | db.generate_block() + p2p().broadcast_block() |
produced |
Emergency consensus active:
- Emergency master (has
emergency-private-key+ committee in schedule): proceeds unconditionally. - Slave: requires
get_slot_time(1) >= now(chain not stale) before producing.
Normal mode (HF12+):
- Participation ≥ 33%: healthy network; sync check via
get_slot_time(1). - Participation < 33%: distressed network; participation vs
required-participationthreshold applies. enable-stale-production=true: bypasses both participation and sync checks.
Pre-HF12: Simple sync check via get_slot_time(1).
When a competing block exists at head_block_num + 1:
- Vote-weighted comparison (HF12+):
compare_fork_branches()computes total SHARES delegated to each branch. If our branch is heavier, proceed and remove the competing block. If tied or lighter, defer. - Stuck-head timeout: After
fork-collision-timeout-blocksconsecutive deferrals (default 21 blocks = 63 seconds before HF16, 21 seconds after), the competing block is removed and production resumes. This handles dead-fork blocks from disconnected peers.
Emergency mode: Any competing block triggers a defer; the vote-weight path is not taken.
Before each production attempt (after HF12 safety checks), the plugin walks the last 21 blocks in fork_db. If all 21 were produced by the node's own configured validators, the node is isolated on a minority fork.
- Default action: Call
p2p().resync_from_lib()— pop blocks to LIB, reset fork DB, re-initiate P2P sync, reconnect seed nodes. Returnsminority_fork. - With
enable-stale-production=true: Log a warning, continue producing. Note: at ≥33% participation this override is auto-cleared every block, so on a single-operator fork it does not stop the detector — usedisable-minority-fork-detectioninstead. - With
disable-minority-fork-detection=true: Both the standard and DLT detection paths are skipped entirely, and the flag is never auto-cleared. For single-operator testnet/forks where "21 blocks all ours" is the healthy steady state. Never enable on a real public network — it removes the isolation guard. - Skipped when: Emergency consensus is active (committee blocks would always match our configured set). A DLT-specific slave isolation check replaces it in emergency mode.
If get_slot_at_time(now) returns 0 (NTP behind chain time), the _slot_zero_streak counter increments:
| Streak | Time | Action |
|---|---|---|
| 3 | ~750 ms | Warning |
| 10 | ~2.5 s | Force NTP resync |
| 60 | ~15 s | Prolonged stall warning |
| 120 | ~30 s | Critical error |
The counter resets on any non-zero slot result.
If the node has ever produced a block and should_be_producing is true (derived from live chain state: participation ≥ 33% or emergency consensus active with our key), but no block has been produced for:
- Emergency master: 60 seconds
- Regular validator: 180 seconds
The watchdog fires every 30 seconds and logs a diagnostic. If recovery conditions are met (head advancing in the last 30 s, not syncing, has peer connections, has non-zero signing keys on-chain), it force-clears blocking conditions:
- Clears
_minority_fork_recoveringflag. - Calls
p2p().clear_catchup_flag()— clears the P2P post-pause catchup flag. - Calls
chain().clear_syncing()— clears the chain sync flag.
Production resumes automatically on the next tick.
Connected to database::applied_block. Runs for every incoming block.
This deferred callback does not refresh the signing-key snapshot.
Block headers must use the key committed at the current head, not a pending
validator_update_operation key. The plugin seeds configured validators' keys at
startup, then replaces the complete snapshot on database::committed_chain_state.
That signal runs synchronously under the database write lock after successful
_push_block() and memory checking, inside the callback that temporarily removes
pending transactions. It fires before those transactions are restored. Signal
callbacks must not yield or acquire database locks.
Reading all configured validators at this boundary also captures rotations
reverted by a fork switch, including cases where the adopted blocks contain no
rotation at all. Applying only incoming block operations cannot undo such a
rotation; reading the database in deferred applied_block instead sees pending
state. The snapshot stores both head height and ID, and removes validators absent
from the committed index. Visible null keys still disable production; a committed
null key is not replaced by a pending re-enable.
Offline regression (requires Python 3 and a C++14 compiler):
python3 tests/validator/test_signing_cache.py
# Negative control against the pre-fix source; expected to exit 1:
python3 tests/validator/test_signing_cache.py --source-ref d4fd4bcdThe test extracts and compiles the real plugin helpers with a controlled database
fixture, and guards the source hook placement and connection lifecycle. It does
not replace a native chain integration run. The companion scenario
tests/validator/test_committed_chain_state.cpp is included in the testnet
consensus_sim_tests target and verifies that real push_block() exposes committed keys in
the synchronous signal and restored pending keys in deferred applied_block.
When block_num > prev_num + 1 (gap in block stream), the handler determines whether our validator was scheduled for any of the missed slots and logs full diagnostic state (production flags, NTP offset, sync status, signing key status, next slot time).
When emergency consensus is active, the emergency master may blank our validator's signing key and produce committee blocks in our scheduled slots. The handler tracks this via _slot_hijack_count. Resets when one of our own validators produces a block.
Returns true if a locally-controlled validator is scheduled to produce in the next 4 slots (~12 seconds). The snapshot plugin calls this before scheduling a snapshot to defer if production is imminent.
Returns true when:
emergency-private-keyis configured (CHAIN_EMERGENCY_VALIDATOR_ACCOUNTin_validators).- The "committee" account is in the current validator schedule.
Only nodes where both conditions hold should produce solo during emergency mode; others are followers and must sync first.
Returns true if emergency-private-key is configured, regardless of the current schedule. Used in P2P hello messages (has_emergency_key field).
Returns a compact diagnostic string:
validator[skip_flags=0x0 catching_up=0 head=#79881136 last_prod=45s_ago minority_rcv=0 slot_hijacks=0]
Included in P2P FORWARD stagnation logs when the node is stuck with no peer ahead.
- Never produce in DLT mode while syncing — creates blocks on a stale head, causing fork oscillation.
- Never produce while snapshot is in progress — write-lock deadlock.
- Never produce if the slot is already filled — creates a micro-fork.
- Emergency master must always produce — it is the sole block producer; waiting would deadlock.
- Slaves must sync before producing in emergency mode — producing on stale head = minority fork.
- Participation < 33% stops production — network partition guard (overridable).
- 21 consecutive own-validator blocks → rollback to LIB — minority fork recovery.
- Committed signing keys are refreshed at each successful push boundary — other production state is read fresh; emergency mode can activate/deactivate any block.
| Symptom | Check |
|---|---|
not_synced logs |
DLT sync active or snapshot in progress — wait; watchdog will auto-clear if stuck |
not_time_yet repeated |
NTP behind chain time; check _slot_zero_streak warnings and NTP offset |
not_my_turn on our slot |
Signing key blanked on-chain; send validator_update_operation to restore |
no_private_key |
Config missing private-key for the signing key that's registered on-chain |
low_participation |
Network participation < 33%; check peer connectivity or set enable-stale-production=true |
fork_collision |
Competing block at next height; wait for vote-weight resolution or 21-deferral timeout |
minority_fork |
Isolated; plugin auto-resyncs to LIB. On a single-operator fork this loops — set disable-minority-fork-detection=true |
| Watchdog fires repeatedly | Sync or catchup flag stuck; watchdog auto-clears if head is advancing |
SLOT-HIJACK logs |
Emergency master blanked our key; restore via validator_update_operation |
See also: Validator Guard, Fair-DPOS, Emergency Consensus, Block Processing.