Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions content/validators/changelog/v0.5.13.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## v0.5.13

### Bug fixes

- Self-recover stalled sync gate
- Boot degraded when a module fails
- Stop dealing turns to non-actors
2 changes: 1 addition & 1 deletion content/validators/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ node:
mode: "validator"
# Address of the ValidatorWallet contract (required for validator mode)
validatorWalletAddress: ""
# Address of the operator that performs consensus duties for the ValidatorWallet
# Address of the operator that owns the ValidatorWallet
operatorAddress: ""
admin:
port: 9155
Expand Down
22 changes: 22 additions & 0 deletions content/validators/greybox-setup-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,4 +174,26 @@ After editing the YAML, restart the LLM module (see "Updating Greybox on a Runni
**All models exhausted error:**
- OpenRouter may be down or your key is invalid
- Check your key at https://openrouter.ai/settings/keys
- Run `task node:doctor` β€” the "LLM Providers" section probes each enabled backend and reports which ones actually answer, rather than only whether a key variable is set
- Fallback providers (heurist, ionet) also need valid keys if you want fallback to work

## No usable LLM backend: the node starts DEGRADED

A node with no working LLM backend no longer refuses to start. It boots, syncs and
stays in the validator set, but it votes **Timeout** on every transaction that
needs an LLM, logs a loud alert naming the broken module, and reports
`genvm_modules.llm.state = "degraded"` on ops `/health` while `status` stays
`"up"`. That is better than the old behaviour β€” the node used to exit about 75
seconds after start and then be banned for idleness β€” but it is **not** a working
validator: Timeout votes are penalized. Treat it as a failure to fix, not a state
to run in. The same applies to the web module when the webdriver is unreachable.

Recovery is automatic once a provider answers again β€” the node learns from its own
transaction traffic and needs no restart. The **one exception** is a key added to
`.env`: the node reads its environment at exec, so a running process cannot pick
that up. Restart it. Editing `genvm-module-llm.yaml` does not require a restart.

**Configure more than one provider.** A backend only enters the fallback chain if
its key is set, so a node set up with OpenRouter alone has no fallback: when that
provider goes down or is decommissioned, the chain exhausts immediately. `doctor`
warns when only one backend is usable.
39 changes: 39 additions & 0 deletions pages/api-references/genlayer-node.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1126,6 +1126,18 @@ curl -X GET http://localhost:9153/health
"status": "up",
"node_version": "v1.2.3",
"protocol_version": "consensus-v2.1",
"genvm_modules": {
"llm": {
"state": "healthy",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T10:30:45Z"
},
"web": {
"state": "healthy",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T10:30:45Z"
}
},
"checks": {
"test-check": {
"status": "up",
Expand All @@ -1140,12 +1152,39 @@ curl -X GET http://localhost:9153/health
- `status` (string): Overall health status - "up" if all checks pass, "down" if any check fails
- `node_version` (string): The version of the GenLayer node software
- `protocol_version` (string): The consensus protocol version
- `genvm_modules` (object): Per-module GenVM health β€” whether this node can currently do LLM and web work. Reported for `llm` and `web`.
- `state` (string): `healthy`, `suspect`, `degraded`, or `unknown`. `suspect` means one failure has been observed and is not yet treated as confirmed; `unknown` means the module has not been exercised yet.
- `timestamp` (string, optional): ISO 8601 time the verdict was last updated. Absent until the module is first observed.
- `last_ok` (string, optional): ISO 8601 time the module last answered successfully. Absent if it never has.
- `detail` (string, optional): the module's own error text. Present only while not healthy.
- `checks` (object, optional): Map of individual health check results
- `[check-name]` (object): Results for a specific health check
- `status` (string): Check status - "up" or "down"
- `timestamp` (string): ISO 8601 timestamp of when the check was performed
- `error` (string, optional): Error message if check failed

**GenVM module health does not affect `status`.**

A node with a broken module is still booted, synced and voting β€” it votes Timeout on the transactions that need that module, which is the designed degraded behaviour rather than an outage. Failing the endpoint would pull a working validator out of load balancers and page an operator about a node behaving as intended. So `genvm_modules` is reported alongside `checks` rather than as one of them, and `status` stays `up`:

```json
{
"status": "up",
"genvm_modules": {
"llm": {
"state": "degraded",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T04:12:03Z",
"detail": "provider unreachable β€” voting Timeout on transactions that need an LLM"
},
"web": { "state": "healthy", "timestamp": "2024-01-15T10:30:45Z", "last_ok": "2024-01-15T10:30:45Z" }
}
}
```

`timestamp` and `last_ok` answer different questions and are both reported. `timestamp` is when the verdict was last updated; `last_ok` is when the module last worked. They match while healthy, and the gap between them while degraded shows how long the outage has run. `timestamp` matters more here than under `checks`: those entries are re-evaluated on every request, whereas module state is folded from real transaction traffic and is not polled while healthy β€” so on a quiet validator a `healthy` verdict can rest on an older observation.


**HTTP Status Codes:**

- `200 OK`: Node is healthy (all checks passing)
Expand Down
39 changes: 39 additions & 0 deletions pages/api-references/genlayer-node/ops/health.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,18 @@ curl -X GET http://localhost:9153/health
"status": "up",
"node_version": "v1.2.3",
"protocol_version": "consensus-v2.1",
"genvm_modules": {
"llm": {
"state": "healthy",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T10:30:45Z"
},
"web": {
"state": "healthy",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T10:30:45Z"
}
},
"checks": {
"test-check": {
"status": "up",
Expand All @@ -35,12 +47,39 @@ curl -X GET http://localhost:9153/health
- `status` (string): Overall health status - "up" if all checks pass, "down" if any check fails
- `node_version` (string): The version of the GenLayer node software
- `protocol_version` (string): The consensus protocol version
- `genvm_modules` (object): Per-module GenVM health β€” whether this node can currently do LLM and web work. Reported for `llm` and `web`.
- `state` (string): `healthy`, `suspect`, `degraded`, or `unknown`. `suspect` means one failure has been observed and is not yet treated as confirmed; `unknown` means the module has not been exercised yet.
- `timestamp` (string, optional): ISO 8601 time the verdict was last updated. Absent until the module is first observed.
- `last_ok` (string, optional): ISO 8601 time the module last answered successfully. Absent if it never has.
- `detail` (string, optional): the module's own error text. Present only while not healthy.
- `checks` (object, optional): Map of individual health check results
- `[check-name]` (object): Results for a specific health check
- `status` (string): Check status - "up" or "down"
- `timestamp` (string): ISO 8601 timestamp of when the check was performed
- `error` (string, optional): Error message if check failed

**GenVM module health does not affect `status`.**

A node with a broken module is still booted, synced and voting β€” it votes Timeout on the transactions that need that module, which is the designed degraded behaviour rather than an outage. Failing the endpoint would pull a working validator out of load balancers and page an operator about a node behaving as intended. So `genvm_modules` is reported alongside `checks` rather than as one of them, and `status` stays `up`:

```json
{
"status": "up",
"genvm_modules": {
"llm": {
"state": "degraded",
"timestamp": "2024-01-15T10:30:45Z",
"last_ok": "2024-01-15T04:12:03Z",
"detail": "provider unreachable β€” voting Timeout on transactions that need an LLM"
},
"web": { "state": "healthy", "timestamp": "2024-01-15T10:30:45Z", "last_ok": "2024-01-15T10:30:45Z" }
}
}
```

`timestamp` and `last_ok` answer different questions and are both reported. `timestamp` is when the verdict was last updated; `last_ok` is when the module last worked. They match while healthy, and the gap between them while degraded shows how long the outage has run. `timestamp` matters more here than under `checks`: those entries are re-evaluated on every request, whereas module state is folded from real transaction traffic and is not polled while healthy β€” so on a quiet validator a `healthy` verdict can rest on an older observation.


**HTTP Status Codes:**

- `200 OK`: Node is healthy (all checks passing)
Expand Down
8 changes: 8 additions & 0 deletions pages/validators/changelog.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## v0.5.13

### Bug fixes

- Self-recover stalled sync gate
- Boot degraded when a module fails
- Stop dealing turns to non-actors

## v0.5.12

### Misc
Expand Down
6 changes: 3 additions & 3 deletions pages/validators/setup-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -216,18 +216,18 @@ One full node can be shared between multiple validators. The optimal validator-t
You should see a list like this

```sh
v0.5.13
v0.5.12
v0.5.11
v0.5.10
v0.5.9
v0.5.8
```

Typically, you will want to run the latest version

2. Download the packaged application
```sh copy
export version=v0.5.12 # set your desired version here
export version=v0.5.13 # set your desired version here
wget https://storage.googleapis.com/gh-af/genlayer-node/bin/amd64/${version}/genlayer-node-linux-amd64-${version}.tar.gz
```
3. Extract the node software
Expand Down Expand Up @@ -311,7 +311,7 @@ node:
mode: "validator"
# Address of the ValidatorWallet contract (required for validator mode)
validatorWalletAddress: ""
# Address of the operator that performs consensus duties for the ValidatorWallet
# Address of the operator that owns the ValidatorWallet
operatorAddress: ""
admin:
port: 9155
Expand Down
Loading