diff --git a/docs/llo-bootstrap-bluegreen-deep-dive.md b/docs/llo-bootstrap-bluegreen-deep-dive.md new file mode 100644 index 00000000..20fa045e --- /dev/null +++ b/docs/llo-bootstrap-bluegreen-deep-dive.md @@ -0,0 +1,736 @@ +# LLO Bootstrap & Blue/Green Deep Dive + +> How bootstrapping actually works for LLO, what changes under Blue/Green, what +> `SetStagingConfig` / `SetProductionConfig` / `PromoteStagingConfig` do to each moving part, +> and — critically — **which signals can and cannot tell you whether nodes are peered on a +> *specific* config digest**. +> +> Companion to `llo-protocol-deep-dive.md` (rounds/outcomes/reports) and +> `chainlink/core/notes/LLO-Bootstrap-Connectivity-Troubleshooting.md` (incident runbook). +> +> All line references are against the local `llo-stack` workspace +> (`libocr/`, `chainlink-evm/`, `chainlink/`, `chainlink-data-streams/`) and +> `chainlink-evm@v0.3.4-0.20260623170329-4577ef4ba0ae` for `pkg/relay`. + +--- + +## Table of Contents + +1. [The one-paragraph mental model](#1-the-one-paragraph-mental-model) +2. [What a bootstrap node is (and is not)](#2-what-a-bootstrap-node-is-and-is-not) +3. [The four P2P objects you must keep separate](#3-the-four-p2p-objects-you-must-keep-separate) +4. [Wiring: job spec → libocr](#4-wiring-job-spec--libocr) +5. [Bootstrap startup sequence](#5-bootstrap-startup-sequence) +6. [Blue/Green: slots, not colors](#6-bluegreen-slots-not-colors) +7. [Event-by-event walkthrough](#7-event-by-event-walkthrough) +8. [The Blue-only bootstrap hack (MERC-6839)](#8-the-blue-only-bootstrap-hack-merc-6839) +9. [Answers to the specific questions](#9-answers-to-the-specific-questions) +10. [How to check if a bootstrap job is working](#10-how-to-check-if-a-bootstrap-job-is-working) +11. [Blue/Green rollout runbook (SetStagingConfig)](#11-bluegreen-rollout-runbook-setstagingconfig) +12. [Signal reference: logs, metrics, code map](#12-signal-reference-logs-metrics-code-map) + +--- + +## 1. The one-paragraph mental model + +A bootstrap node is a **rendezvous point for peer address discovery**, nothing else. It reads +the DON's on-chain config, extracts the *oracle peer ID set*, registers that set as a +"group" with the discovery layer, and then relays signed address announcements between +members of that group. It never sees stream data, never runs the LLO plugin, never +transmits, and — contrary to a common belief — **never pushes OCR config to oracles**. +Oracles read config from the chain themselves. Under Blue/Green, an oracle node runs **two +independent OCR3 instances** (Blue and Green), each with its own config digest, its own +discovery group and its own message streams — but they share one TCP connection per peer and +one process-wide set of P2P metrics. That sharing is the root of nearly every +"is staging actually peered?" confusion. + +--- + +## 2. What a bootstrap node is (and is not) + +libocr states the role outright: + +```go +// Bootstrapper connects to a particular feed and listens for config changes, +// but does not participate in the protocol. It merely acts as a bootstrap node +// for peer discovery. +``` +`libocr/offchainreporting2plus/bootstrapper.go:35` + +`bootstrapperV2` has **no message loop at all** — its entire body is a constructor that logs +`BootstrapperV2: Initialized`, a `Start()` that logs `BootstrapperV2: Started listening`, and +a `Close()` that releases the group registration +(`libocr/networking/bootstrapper_v2.go:44-80`). All actual work happened in +`concretePeerV2.register()` (`libocr/networking/peer_v2.go:157`), which calls +`discoverer.AddGroup(configDigest, oracles, bootstrappers)`. + +| Bootstrap node **does** | Bootstrap node **does not** | +|---|---| +| Read configurator events via the log poller DB | Call the RPC directly per poll | +| Register the oracle peer ID set as a discovery group | Run OCR3 consensus, observations, or reports | +| Accept inbound ragep2p connections from those peer IDs | Accept connections from peers outside the group | +| Relay signed announcements (`ragedisco/v1`) between group members | Send OCR config to oracles over P2P | +| Answer `rageping` probes | Transmit reports anywhere | + +**Correction to a widespread claim:** you will see docs/AI answers describing bootstrap as a +"config server" that "broadcasts config to oracle peers over P2P". It doesn't. Every oracle +has its own `ContractConfigTracker` polling the same configurator contract +(`chainlink/core/services/llo/delegate.go:175`). Bootstrap's config read exists *only* to +learn which peer IDs are allowed in the group. This matters operationally: a bootstrap stuck +on a stale digest does not give oracles stale config — it gives them a **wrong allowlist**, +which manifests as `Received incoming connection from an unknown peer, closing` +(`libocr/ragep2p/ragep2p.go:769`). + +--- + +## 3. The four P2P objects you must keep separate + +Almost every Blue/Green observability trap comes from conflating these. + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ 1. CONNECTION (ragep2p) — one TCP/TLS conn per (peer_id, remote) │ +│ Shared by every config digest. Metrics: ragep2p_peer_conn_* │ +├──────────────────────────────────────────────────────────────────────────┤ +│ 2. GROUP (ragedisco) — one per configDigest │ +│ Defines: who we accept, whose announcements we relay to whom │ +│ Registered by AddGroup(); metrics are UNIONS across all groups │ +├──────────────────────────────────────────────────────────────────────────┤ +│ 3. ANNOUNCEMENT (ragedisco) — one per peer, signed, counter-versioned │ +│ Process-wide map bestAnnouncement[peerID]. NOT per digest. │ +├──────────────────────────────────────────────────────────────────────────┤ +│ 4. STREAM (ragep2p) — per (peer, streamName) │ +│ "ragedisco/v1", "ping-pong-…", and "ocr/" │ +│ ← THE ONLY PER-DIGEST OBJECT. And it has no Prometheus metrics. │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +Stream naming: `fmt.Sprintf("ocr/%s", cd)` (`libocr/networking/ocr_endpoint_v2.go:145`); OCR3.1 +adds a second stream `ocr//priority=low` +(`libocr/networking/ocr_endpoint_v3.go:596`). + +### 3.1 Group semantics (the rules that decide discovery) + +From `libocr/networking/ragedisco/discovery_protocol.go`: + +| Rule | Code | Consequence | +|---|---|---| +| An announcement from peer B is **rejected** unless B is an oracle in ≥1 of *our* groups | `:544` — `"peer %s is not an oracle in any of our jobs"` | A node absent from every group we track is invisible to us | +| We forward B's announcement to every peer in every group where B is an oracle | `lockedAllowedPeers`, `:213` | Relaying is decided by **the relayer's own** group membership | +| Adding a group opens `ragedisco/v1` streams to all its members | `:285` → `connectivityAdd` | Group membership ⇒ known peer ⇒ inbound connections accepted | +| Bootstrapper addresses come from **local config**, not gossip, and win over announcements | `FindPeer`, `:458-474` | `p2pv2Bootstrappers` typos are never self-healing | +| Removing a group drops peers only if they're in no other group | `removeGroup`, `:397-455` | Blue+Green overlap keeps connections alive across config switches | + +--- + +## 4. Wiring: job spec → libocr + +There is **no LLO-specific bootstrap delegate**. LLO-ness enters via `providerType = "llo"`. + +```mermaid +flowchart TB + subgraph boot["BOOTSTRAP NODE — type = \"bootstrap\""] + BJ["job spec
contractID = configurator
relayConfig{chainID, fromBlock,
lloConfigMode, lloDonID, providerType}"] + BD["ocrbootstrap/delegate.go
(generic, all plugins)"] + BCP["chainlink-evm/pkg/relay/llo_config_provider.go
lloConfigProvider"] + BPOLL["Blue + Green LLOConfigPoller
(both started)"] + BTRACK["ContractConfigTracker()
returns cps[0] = BLUE ONLY ⚠"] + BBS["libocr Bootstrapper
→ AddGroup(digest, oracles)"] + end + + subgraph oracle["ORACLE NODE — type = \"offchainreporting2\", pluginType = \"llo\""] + OJ["job spec
+ p2pv2Bootstrappers
+ pluginConfig"] + OD["ocr2/delegate.go → newServicesLLO"] + OPROV["chainlink-evm/pkg/relay/llo_provider.go
NewLLOProvider"] + OPOLL["Blue + Green LLOConfigPoller"] + OLD["core/services/llo/delegate.go
one OCR3 Oracle PER tracker"] + OB["OCR3 Oracle #0 = Blue
group + ocr/<blueDigest>"] + OG["OCR3 Oracle #1 = Green
group + ocr/<greenDigest>"] + PLUG["chainlink-data-streams/llo
ReportingPlugin (per instance)"] + end + + LPDB[("Log Poller DB
(chain-wide, one per chainID)")] + CONF["Configurator contract
ProductionConfigSet / StagingConfigSet /
PromoteStagingConfig"] + + CONF -->|logs| LPDB + BJ --> BD --> BCP --> BPOLL --> LPDB + BCP --> BTRACK --> BBS + OJ --> OD --> OPROV --> OPOLL --> LPDB + OD --> OLD --> OB & OG + OB --> PLUG + OG --> PLUG + BBS -. "discovery relay only
(NO config transfer)" .- OB +``` + +Key code: + +| Concern | Location | +|---|---| +| Bootstrap service assembly | `chainlink/core/services/ocrbootstrap/delegate.go` (`ocr.NewBootstrapper`) | +| `providerType="llo"` routing | `chainlink-evm/pkg/relay/evm.go` → `newLLOConfigProvider` | +| Bootstrap provider (Blue-only tracker) | `chainlink-evm@…/pkg/relay/llo_config_provider.go:38-43` | +| Blue+Green poller construction | `chainlink-evm@…/pkg/relay/llo_provider.go:349-375` (`cps = []{blueCP, greenCP}`) | +| Config poller (event → ContractConfig) | `chainlink-evm/pkg/llo/config_poller.go:148-212` | +| One OCR3 oracle per tracker | `chainlink/core/services/llo/delegate.go:147-216` | +| Plugin lifecycle stage | `chainlink-data-streams/llo/v30/plugin_outcome.go:26-105` | + +Bootstrap jobs do **not** set `p2pv2Bootstrappers` — they *are* the bootstrap peer. +The `[relayConfig]` fields are all effectively required; omitting `lloConfigMode`, +`lloDonID`, or `providerType` yields a job that starts but never loads config. + +--- + +## 5. Bootstrap startup sequence + +```mermaid +sequenceDiagram + participant JS as job/spawner.go + participant D as ocrbootstrap.Delegate + participant CP as lloConfigProvider (Blue+Green pollers) + participant LP as Log Poller DB + participant MB as managed.RunManagedBootstrapper + participant DISC as ragedisco discoveryProtocol + + JS->>D: ServicesForSpec(job) + D->>CP: relayer.NewConfigProvider(providerType="llo") + Note over CP: registers log poller filter
(ProductionConfigSet, StagingConfigSet,
PromoteStagingConfig; Topic2 = donID) + CP->>LP: Replay(fromBlock) — ONLY if job is brand new + D->>MB: ocr.NewBootstrapper(ContractConfigTracker = cps[0] /*Blue*/) + loop every contractConfigTrackerPollInterval + MB->>CP: LatestConfigDetails() + CP->>LP: FilteredLogs(addr, sigs, donID, block ≥ fromBlock) + alt no matching logs + CP-->>MB: zero configDigest + Note over MB: "TrackConfig: LatestConfigDetails()
returned a zero configDigest" — forever + else config found + CP-->>MB: digest + oracle set ("LatestConfig fetched") + MB->>DISC: AddGroup(digest, oracles, bootstrappers) + Note over DISC: "Ragep2pDiscoverer: Adding group" + MB->>MB: "BootstrapperV2: Initialized" → "Started listening" + end + end +``` + +Two facts worth burning in: + +1. **`relayConfig.fromBlock` is a DB query lower bound, not a backfill trigger.** Automatic + replay happens only when the job is *newly created* (`opts.New` → `runReplay`, + `llo_config_provider.go:82-95`). Restarting the node or cancel+redeploying a job on a host + whose log poller filter already exists (`Filter already present, no-op`) does **not** + re-index history. You must `POST /v2/replay_from_block/`. +2. **One group at a time per bootstrap job.** `RunManagedBootstrapper` tears down the old + bootstrapper (and thus `RemoveGroup`) and builds a new one on every config change + (`libocr/offchainreporting2plus/internal/managed/managed_bootstrapper.go:31-58`). + +--- + +## 6. Blue/Green: slots, not colors + +**Blue and Green are two *slots*. `isGreenProduction` decides which slot is currently +"production".** Neither slot is inherently production or staging. + +The whole rule is one line, evaluated per event +(`chainlink-evm/pkg/llo/config_poller.go:180` and `:198`): + +```go +isProduction := (cp.instanceType != InstanceTypeBlue) == event.IsGreenProduction +// ProductionConfigSet: adopt if isProduction +// StagingConfigSet: adopt if !isProduction +``` + +Truth table: + +| `isGreenProduction` | Blue slot is | Green slot is | `ProductionConfigSet` lands in | `StagingConfigSet` lands in | +|---|---|---|---|---| +| `false` (initial) | **production** | staging | Blue | Green | +| `true` (after 1 promote) | staging | **production** | Green | Blue | +| `false` (after 2 promotes) | **production** | staging | Blue | Green | + +`PromoteStagingConfig` is **not** consumed by the config poller at all — it only flips the +contract's `isGreenProduction` flag, which then appears in the payload of *subsequent* +config-set events. The digest does not change on promotion; only the **role** changes. +The promote event is consumed by a different component, the `ShouldRetireCache` +(`chainlink-evm/pkg/llo/should_retire_cache.go:53,114`), which tells the outgoing instance to +retire. + +### 6.1 Instance ↔ lifecycle stage + +The *plugin* learns whether it is staging from the config itself, not from the slot: + +```go +// plugin_outcome.go:26-33 (SeqNr == 1, the cornerstone outcome) +if p.PredecessorConfigDigest == nil { + lifeCycleStage = protocol.LifeCycleStageProduction +} else { + lifeCycleStage = protocol.LifeCycleStageStaging +} +``` + +A staging instance stays in `staging` until it observes a valid **attested retirement report** +from its predecessor, then flips itself to `production` +(`plugin_outcome.go:72-79`). The old production instance flips to `retired` once >F nodes +observe `ShouldRetire` for its digest (`plugin_outcome.go:84-86`), which they learn from the +`PromoteStagingConfig` log. That handshake — not the chain event — is what makes the cutover +gapless. + +```mermaid +sequenceDiagram + participant Chain as Configurator + participant Blue as Blue instance (production) + participant Green as Green instance (staging) + + Chain->>Green: StagingConfigSet(digest_G, predecessor = digest_B) + Note over Green: LifeCycleStage = staging
runs full OCR3 rounds, does not own the feed + Chain->>Blue: PromoteStagingConfig (flips isGreenProduction=true) + Note over Blue: ShouldRetireCache → ShouldRetire(digest_B) = true + Note over Blue: >F votes → LifeCycleStage = retired
emits attested RetirementReport + Blue-->>Green: RetirementReport (via RetirementReportCache) + Note over Green: sees valid predecessor retirement
→ LifeCycleStage = production +``` + +--- + +## 7. Event-by-event walkthrough + +Assume steady state: `isGreenProduction = false`, Blue holds production digest `B1`, +Green holds nothing. + +### 7.1 `SetStagingConfig` (your imminent rollout) + +| Component | What happens | +|---|---| +| Contract | Emits `StagingConfigSet(donID, digest=G1, …, isGreenProduction=false)` | +| Oracle Blue poller | `isProduction = true` → ignores staging event. Keeps `B1`. | +| Oracle Green poller | `isProduction = false` → adopts `G1`. Logs `LatestConfig fetched … instanceType=Green` | +| Oracle Green OCR3 | `runWithContractConfig: switching between configs` → new endpoint → `AddGroup(G1, oracles_G1)` → opens `ocr/G1` streams → `OCREndpointV2: Initialized configDigest=G1` | +| Oracle plugin | New plugin instance with `PredecessorConfigDigest = B1` ⇒ `LifeCycleStage = staging`; starts running real rounds | +| **Bootstrap node** | **Nothing happens.** Its tracker is Blue-only; the Blue slot is still `B1`. No new group, no log line. | +| Production traffic | Untouched. Blue keeps producing and transmitting. | + +The critical inference: **the bootstrap node plays no part in a `SetStagingConfig` unless the +staging config introduces peer IDs that are not already in the Blue-slot config.** If the +node set is unchanged, staging peering "just works" because the peers, connections and +announcements already exist — only new `ocr/G1` streams are layered on top. + +If the staging config **adds a new node**, that node is: +- not in the bootstrap's group → bootstrap logs `unknown peer, closing` and refuses it; +- not an oracle in any group the bootstrap tracks → the bootstrap will not relay its + announcement (`discovery_protocol.go:213`, `:544`); +- unable to learn any oracle address, because its only static address is the bootstrap. + +Existing oracles *do* register it (it's in their Green group), so they will accept it — but +they can't dial it until they receive its announcement, and the only relay path is the +bootstrap. **Result: a staging-only new node cannot join until the bootstrap tracks a config +containing it.** Plan node-set changes accordingly (see §11). + +### 7.2 `SetProductionConfig` + +| Component | What happens (`isGreenProduction = false`) | +|---|---| +| Oracle Blue poller | Adopts new production digest `B2`; Blue OCR3 switches config, removes group `B1`, adds group `B2`, opens `ocr/B2` streams | +| Oracle Green poller | Ignores it | +| Bootstrap | Blue tracker sees `B2` → `runWithContractConfig: switching between configs` → old bootstrapper closed (`RemoveGroup(B1)`) → `BootstrapperV2: Initialized configDigest=B2` | +| Plugin | New instance, `PredecessorConfigDigest == nil` ⇒ starts directly in `production` | + +Note there is **no staging/retirement handshake** here — a direct `SetProductionConfig` is a +hard cutover of the production instance. Expect a short reporting gap while the new instance +reaches its first commit (this shows up as one large report range, not a data gap; see +`llo-protocol-deep-dive.md` §5). + +**Caveat:** if `isGreenProduction = true` at the time, `SetProductionConfig` targets the +**Green** slot — and the Blue-only bootstrap will *not* follow it (§8). + +### 7.3 `PromoteStagingConfig` + +| Component | What happens | +|---|---| +| Contract | Flips `isGreenProduction` false→true. Digests unchanged. | +| Config pollers | Nothing adopts a new config — no `ConfigSet` event was emitted. Blue keeps `B1`, Green keeps `G1`. | +| `ShouldRetireCache` | Sees the promote log; `ShouldRetire(B1)` becomes true | +| Blue plugin | >F nodes observe `ShouldRetire` → `LifeCycleStage = retired`, emits attested retirement report, stops producing | +| Green plugin | Consumes predecessor retirement report → `LifeCycleStage = production` | +| Bootstrap | Nothing. Still serving group `B1` — which is now the **retired** config's node set. | +| Slot semantics | From now on, `StagingConfigSet` lands in **Blue**, `ProductionConfigSet` lands in **Green** | + +--- + +## 8. The Blue-only bootstrap hack (MERC-6839) + +```go +func (l *lloConfigProvider) ContractConfigTracker() ocrtypes.ContractConfigTracker { + // FIXME: Only return Blue for now. This is a hack to make the bootstrap + // job work, needs to support multiple config trackers here + // MERC-6839 + return l.cps[0] +} +``` +`chainlink-evm@…/pkg/relay/llo_config_provider.go:38-43` + +Both pollers are constructed and started; only Blue is exposed to libocr. Implications, +ordered by how likely they are to bite you: + +| # | Implication | Practical impact | +|---|---|---| +| 1 | Bootstrap group = **whatever config sits in the Blue slot**, which is production only while `isGreenProduction == false` | After an odd number of promotions, the bootstrap's allowlist is the *staging/retired* slot's node set | +| 2 | Bootstrap never registers a group for a staging digest set in the Green slot | Staging-only peer IDs are rejected and their announcements are never relayed (§7.1) | +| 3 | Bootstrap emits **no log line** when you `SetStagingConfig` into Green | Absence of bootstrap activity is expected, not a fault — do not chase it | +| 4 | With overlapping node sets (the normal case) none of this is visible | Which is exactly why a node-set change during blue/green is the dangerous scenario | + +Mitigation available today: keep the DON's peer ID set identical between production and +staging configs, and introduce/remove nodes via a `SetProductionConfig` (or a promotion) that +lands in the slot the bootstrap tracks, *before* relying on them in staging. + +--- + +## 9. Answers to the specific questions + +### Q1. How does bootstrapping work? + +Discovery-only rendezvous. Oracle dials `peerID@host:port` from `p2pv2Bootstrappers` +(static, never learned via gossip), opens a `ragedisco/v1` stream, and exchanges signed +address announcements. The bootstrap relays announcements between members of the group it +registered from on-chain config. Once an oracle knows another oracle's address, all further +traffic is direct peer-to-peer; the bootstrap is not in the data path. See §2, §3. + +### Q2. How does it work for the LLO plugin? + +Identically — the LLO plugin has nothing to do with bootstrapping. There is no LLO bootstrap +delegate; `providerType = "llo"` only selects a config provider that knows how to parse +configurator v2 blue/green events. The bootstrap node never instantiates the reporting +plugin. See §4. + +### Q3. How does it work under Blue/Green? + +The **oracle** runs two OCR3 instances (`ContractConfigTrackers` = `[Blue, Green]`, one +`ocr2plus.NewOracle` each — `chainlink/core/services/llo/delegate.go:158-216`), each with its +own digest, discovery group and `ocr/` streams. The **bootstrap** runs exactly one +instance and tracks the **Blue slot only**. See §6, §8. + +### Q4. If we do `SetStagingConfig`, does it work? + +Yes, with one caveat. The Green pollers on every oracle adopt the staging digest, the Green +OCR3 instance starts, registers its group and begins real consensus rounds in +`LifeCycleStage = staging`. Production (Blue) is unaffected. The bootstrap node does nothing +and logs nothing. + +**Caveat:** it works *because* the staging node set is already peered via the production +group. If your staging config introduces a peer ID that is not in the Blue-slot config, that +node cannot join — the bootstrap rejects it (`unknown peer, closing`) and will not relay its +announcement. See §7.1. + +### Q5. If we do `SetProductionConfig`, does it work? + +Yes. It replaces the config in whichever slot is currently production, and that instance +hard-switches: old group removed, new group added, new plugin instance starting directly in +`production` (no retirement handshake, so expect one enlarged report range at cutover). +The bootstrap follows it **only if the production slot is the Blue slot** +(`isGreenProduction == false`). After an odd number of promotions, `SetProductionConfig` +targets Green and the bootstrap keeps serving the old Blue-slot allowlist. See §7.2, §8. + +### Q6. Nodes are peered on the production digest — how do we know they're peered on staging? + +**Not from any P2P metric or log**, because none of them are digest-scoped: + +| Signal | Scope | Can it distinguish Blue vs Green? | +|---|---|---| +| `ragep2p_peer_conn_*`, `ragep2p_peer_rawconn_*` | labels: `peer_id`, `remote_peer_id` only (`libocr/ragep2p/metrics.go:51`) | ❌ one shared TCP conn carries both | +| `rageping_*` | labels: `peer_id`, `remote_peer_id`, ping params (`networking/rageping/metrics.go:40-48`) | ❌ | +| `ragedisco_registered_peers` / `_discovered_peers` / `_bootstappers` | one gauge per process, label `peer_id` only; values are **unions across all groups** (`ragedisco/metrics.go:18`, `discovery_protocol.go:237-243`) | ❌ (but see the union trick below) | +| `DiscoveryProtocol: Status report` (`peersToDetect` / `peersUndetected`) | union over `numGroupsByOracle` (`discovery_protocol.go:188-202`) | ❌ | +| `ocr3_epoch`, `ocr3_committed_sequence_number`, `ocr3_*` | **no digest label at all**, and Blue+Green register identical collectors under the same `job_name` — the second registration fails (`RegisterOrLogError`, `libocr/internal/metricshelper`), so only **one** instance's series exists | ❌ actively misleading | +| `ocr3_reporting_plugin_status{plugin="llo", configDigest=…}` | **labelled by configDigest** (`chainlink/core/services/ocr3/promwrapper/types.go:62-66`) | ✅ | +| Logs `OCREndpointV2/V3: Initialized` + `Started listening`, `Ragep2pDiscoverer: Adding group` | carry `configDigest`; oracle loggers also carry `instanceType=Blue/Green` | ✅ | +| LLO telemetry / reports (observation & outcome telemetry carry the digest — `plugin_observation.go:158`) | per instance | ✅ | + +So use, per node: + +```promql +# 1. Staging plugin instance exists and is running +ocr3_reporting_plugin_status{plugin="llo", configDigest=""} == 1 + +# 2. Union grew as expected (only informative when the staging set adds nodes) +ragedisco_registered_peers{peer_id=""} # should equal |blue ∪ green| +ragedisco_discovered_peers{peer_id=""} # should converge to the same number +``` + +plus logs scoped to the digest: + +```logql +{host=""} |= "OCREndpointV2: Initialized" |= "" +{host=""} |= "Ragep2pDiscoverer: Adding group" |= "" +{host=""} |= "instanceType=Green" |= "LatestConfig fetched" +``` + +The only *proof* of peering (as opposed to configuration) is **consensus progress on the +staging instance**: the Green instance committing sequence numbers and producing staging +reports/telemetry. Peering is a means; rounds are the end. If `ocr3_reporting_plugin_status` +for the staging digest is 1 on all nodes but no staging reports appear, you have a config +that loaded but a DON that isn't talking. + +### Q7. Could nodes be un-peered on a new staging config while metrics show "peered"? + +**Yes — this is the expected failure mode, not an edge case.** Concretely: + +1. Every connection-level metric is per-peer, and the production group already holds those + connections open. They will read healthy no matter what happens to staging. +2. `ragedisco` gauges and the `Status report` are unions across groups. If staging has the + same node set as production, **all of these numbers are literally unchanged** by a + `SetStagingConfig` — they cannot report on it. +3. Per-digest activity lives only in `ocr/` **streams**, and ragep2p exports no + per-stream metrics. +4. `ocr3_*` metrics can't help: Blue and Green collide on registration, so you're reading one + instance without knowing which. + +The realistic bad scenario is: staging config adopted by only *some* nodes (log-poller lag, +different `fromBlock`, one node's job not restarted), so the Green instance has fewer than +`2f+1` participants. All P2P dashboards stay green; the staging instance simply never +commits. Detect it with `ocr3_reporting_plugin_status{configDigest=""}` counted +across nodes, and with the absence of staging reports. + +```promql +# How many nodes actually run the staging instance? Must be N (or at minimum 2f+1) +count(ocr3_reporting_plugin_status{plugin="llo", configDigest=""} == 1) +``` + +--- + +## 10. How to check if a bootstrap job is working + +Five checks, in order. Stop at the first failure. + +### 10.1 Job is running + +```bash +# On the bootstrap node +curl -s -H "Authorization: Bearer $CL_API_TOKEN" https:///v2/jobs | jq ' + .data[] | select(.attributes.type=="bootstrap") | + {id, contractID: .attributes.bootstrapSpec.contractID, + relayConfig: .attributes.bootstrapSpec.relayConfig}' +``` +Confirm for your DON: `providerType="llo"`, `lloConfigMode="bluegreen"`, correct `lloDonID`, +`chainID`, and `fromBlock` ≤ the block of the first config-set event for the DON. + +### 10.2 The healthy log sequence exists (per DON, in order) + +``` +ConfigProvider.Blue.LLOConfigPoller Starting / Started +Inserted filter (or: Filter already present, no-op) +LatestConfig fetched instanceType=Blue donID= ← must appear +BootstrapperV2: Initialized configDigest= oracles=[…] +Ragep2pDiscoverer: Adding group configDigest= +BootstrapperV2: Started listening +``` + +Failure signature: + +``` +TrackConfig: LatestConfigDetails() returned a zero configDigest ← repeats every poll +(no "LatestConfig fetched", no "BootstrapperV2: Initialized") +Received incoming connection from an unknown peer, closing ← TCP fine, ragep2p rejects +``` + +`Filter already present, no-op`, `LLOConfigPoller.Blue Started`, and other bootstrap jobs on +the same host being healthy all mean **nothing** about whether this DON's config loaded. + +### 10.3 The oracle set is right + +From `BootstrapperV2: Initialized`, check `oracles=`: +- count matches the on-chain config exactly (a count far above it — e.g. 61 vs 16 — means a + stale group); +- every operator's peer ID you expect to serve is present. + +### 10.4 Oracles are actually connecting + +On the bootstrap host: + +```promql +# Someone is dialing us at all +rate(ragep2p_host_inbound_dials_total[1h]) > 0 + +# This specific oracle is connected and exchanging bytes +rate(ragep2p_peer_conn_read_processed_bytes_total{remote_peer_id=""}[5m]) > 0 +rate(ragep2p_peer_conn_written_bytes_total{remote_peer_id=""}[5m]) > 0 +``` + +Zero for a peer that *is* in `oracles=` ⇒ network/egress problem on that operator. +`unknown peer, closing` for a peer that *should* be in `oracles=` ⇒ bootstrap config problem, +not an operator problem. + +### 10.5 If config never loads: replay, don't restart + +```bash +curl -X POST -H "Authorization: Bearer $CL_API_TOKEN" \ + "https:///v2/replay_from_block/?family=evm&ChainID=" +``` + +Use the job's `fromBlock`, not chain head. Then watch for `LatestConfig fetched` → +`BootstrapperV2: Initialized` within a few minutes. Repeat on **every** bootstrap host serving +the DON — fixing one does not help oracles that dial a different bootstrap peer ID. + +--- + +## 11. Blue/Green rollout runbook (SetStagingConfig) + +### Pre-flight + +- [ ] **Diff the peer ID sets** between the current production config and the intended staging + config. Identical ⇒ low risk. Any addition ⇒ read §7.1 first; the new node will not be + able to join through the bootstrap. +- [ ] Confirm which slot is production: read `isGreenProduction` on the configurator (or infer + it from the last `PromoteStagingConfig`). This tells you whether staging will land in + Green (normal) or Blue (after an odd number of promotions). +- [ ] Confirm the bootstrap nodes are healthy **now** (§10) and record the digest and oracle + count they currently serve. +- [ ] Record baselines per node: `ragedisco_registered_peers`, `ragedisco_discovered_peers`, + and the current production digest. +- [ ] Confirm every oracle node's job has a `fromBlock` low enough that its pollers will see + the new event (they will — it's a new log at chain tip — but a node whose log poller is + lagging or whose filter is missing will silently not adopt). + +### Immediately after `SetStagingConfig` + +Expected within one or two `contractConfigTrackerPollInterval`s, on **every** oracle: + +1. `LatestConfig fetched … instanceType=Green … donID=` with the new digest + (or `instanceType=Blue` if `isGreenProduction=true`). +2. `runWithContractConfig: switching between configs`. +3. `Ragep2pDiscoverer: Adding group configDigest=`. +4. `OCREndpointV2: Initialized configDigest=` → `Started listening`. +5. `Wrapping ReportingPlugin with prometheus metrics reporter configDigest=`. + +Expected on **bootstrap**: nothing at all. That is correct behavior (§8, implication 3). + +### Verification (the part that actually matters) + +```promql +# A. Every node adopted the staging config — this is the single highest-value check +count(ocr3_reporting_plugin_status{plugin="llo", configDigest=""} == 1) +# expect: N (all nodes). Anything < 2f+1 means staging cannot reach consensus. + +# B. Production is undisturbed +ocr3_reporting_plugin_status{plugin="llo", configDigest=""} == 1 + +# C. Connectivity did not regress (necessary, not sufficient — see Q7) +ragedisco_discovered_peers == ragedisco_registered_peers +rate(rageping_timed_out_requests_total[15m]) == 0 + +# D. Only if the staging node set differs: the union grew as expected +ragedisco_registered_peers{peer_id=""} # = |production ∪ staging| peer IDs +``` + +Then confirm the staging instance is **producing**: staging-lifecycle reports/telemetry +arriving at the Data Streams server for the staging digest, and increasing committed sequence +numbers on the Green instance. Config adoption without round progress = not peered. + +### Red flags + +| Observation | Meaning | +|---|---| +| `ocr3_reporting_plugin_status{staging digest}` present on only some nodes | Partial adoption — log poller lag or a node whose job isn't running the Green tracker | +| `peer … is not an oracle in any of our jobs` warnings referencing a *new* staging peer | That node isn't in any group the logging node tracks — expected until every node adopts staging | +| `unknown peer, closing` on bootstrap for a staging-only node | §7.1 — bootstrap cannot admit it; the node is stranded | +| P2P metrics perfectly healthy but no staging reports | The Q7 failure mode. Trust the digest-scoped signals, not the P2P ones. | +| Staging instance never leaves `LifeCycleStage = staging` after promotion | Predecessor retirement report not observed — check `ShouldRetireCache` / `PromoteStagingConfig` indexing | + +### Promotion (later) + +`PromoteStagingConfig` emits no config-set event, so pollers adopt nothing. Watch instead for: +the old instance going `retired` and emitting a retirement report, and the staging instance +flipping to `production`. Remember that after this, the slot↔role mapping inverts and the +Blue-only bootstrap is now tracking the **staging** slot (§8). + +--- + +## 12. Signal reference: logs, metrics, code map + +### 12.1 Log lines by layer + +| Layer | Log | Emitted by | Carries digest? | +|---|---|---|---| +| Config | `LatestConfig fetched` | `chainlink-evm/pkg/llo/config_poller.go:221` | yes + `instanceType`, `donID` | +| Config | `TrackConfig: LatestConfigDetails() returned a zero configDigest` | libocr managed | n/a — the failure state | +| Config | `runWithContractConfig: switching between configs` | `managed/run_with_contract_config.go:99` | yes | +| Discovery | `Ragep2pDiscoverer: Adding group` / `Removing group` | `ragedisco/ragep2p_discoverer.go:260,270` | **yes** | +| Discovery | `DiscoveryProtocol: Status report` | `discovery_protocol.go:197` | no — union | +| Discovery | `DiscoveryProtocol: Replacing our own announcement` | `discovery_protocol.go:665` | no | +| Discovery | `peer … is not an oracle in any of our jobs` | `discovery_protocol.go:545` | no | +| Discovery | `NewStream failed!` / `Write message to peer we don't have a stream open for` | `ragep2p_discoverer.go:169,224` | no | +| Transport | `Received incoming connection from an unknown peer, closing` | `ragep2p/ragep2p.go:769` | no | +| Endpoint | `OCREndpointV2/V3: Initialized`, `Started listening` | `ocr_endpoint_v2.go:115,201`, `v3.go:88,186` | **yes** | +| Endpoint | `No bootstrappers were provided…` | `ocr_endpoint_v2.go:121` | yes | +| Bootstrap | `BootstrapperV2: Initialized` (+`oracles=`), `Started listening` | `bootstrapper_v2.go:48,79` | **yes** | + +### 12.2 Metrics by scope + +| Metric | Labels | Scope | +|---|---|---| +| `ragep2p_peer_conn_*`, `ragep2p_peer_rawconn_*`, `ragep2p_experimental_peer_message_bytes` | `peer_id`, `remote_peer_id` | per peer pair, all digests | +| `ragep2p_host_inbound_dials_total` | `peer_id` | per host | +| `rageping_{sent,received,timed_out}_requests_total`, `rageping_round_trip_latency_seconds` | `peer_id`, `remote_peer_id`, ping params | per peer pair | +| `ragedisco_registered_peers` / `_discovered_peers` / `_bootstappers` | `peer_id` | process-wide **union across groups** | +| `ocr3_epoch`, `ocr3_committed_sequence_number`, `ocr3_sent_observations_total`, `ocr3_included_observations_total`, `ocr3_led_committed_rounds_total` | none (+ `job_name` from the wrapping registerer) | **collides between Blue and Green — treat as untrustworthy on blue/green nodes** | +| `ocr3_reporting_plugin_status` | `chainFamily`, `chainID`, `plugin`, **`configDigest`** | **per instance — the one digest-scoped gauge** | +| `ocr3_reporting_plugin_reports_processed` / `_duration` / `_data_sizes` | `chainFamily`, `chainID`, `plugin`, `function` | not digest-scoped | + +### 12.3 Code map (reading order) + +**Bootstrap path** +1. `chainlink/core/services/job/spawner.go` — how any job starts +2. `chainlink/core/services/ocrbootstrap/delegate.go` — service assembly +3. `chainlink-evm/pkg/relay/evm.go` → `NewConfigProvider` (`providerType="llo"` branch) +4. `chainlink-evm/pkg/relay/llo_config_provider.go` — Blue-only tracker, replay-on-new-job +5. `chainlink-evm/pkg/llo/config_poller.go` — event → `ContractConfig` (the `isProduction` line) +6. `libocr/offchainreporting2plus/internal/managed/managed_bootstrapper.go` — config→group loop +7. `libocr/networking/peer_v2.go` → `register()` → `ragedisco.AddGroup` +8. `libocr/networking/ragedisco/discovery_protocol.go` — the actual discovery rules + +**Blue/Green semantics** +1. `chainlink-evm/pkg/llo/config_poller.go:176-210` + `config_poller_test.go:100-300` + (the tests are the clearest spec of slot flipping) +2. `chainlink-evm/pkg/llo/should_retire_cache.go` — `PromoteStagingConfig` consumption +3. `chainlink/core/services/llo/delegate.go:147-216` — one oracle per tracker +4. `chainlink-data-streams/llo/v30/plugin_outcome.go:20-110` — lifecycle stage transitions +5. `chainlink-data-streams/llo/v30/plugin_observation.go:40-70` — retirement report observation + +--- + +## 13. Cheat sheet + +``` +BOOTSTRAP = DISCOVERY RENDEZVOUS, NOT A CONFIG SERVER + reads on-chain config → oracle peer ID set → ragedisco group + relays signed announcements between group members + no plugin, no consensus, no transmission, no config push + +BLUE/GREEN = TWO SLOTS + A FLAG + isProduction := (instanceType != Blue) == event.IsGreenProduction + isGreenProduction=false → Blue=production, Green=staging (and vice versa) + PromoteStagingConfig only flips the flag; digests never change + +SetStagingConfig → Green pollers adopt; Green OCR3 starts in LifeCycleStage=staging + production untouched; BOOTSTRAP DOES NOTHING (and logs nothing) +SetProductionConfig → production instance hard-switches (no retirement handshake) + bootstrap follows ONLY if production is currently the Blue slot +PromoteStagingConfig → old prod retires + emits retirement report + staging consumes it → becomes production; slot roles invert + +WHAT CAN TELL BLUE FROM GREEN + ✅ ocr3_reporting_plugin_status{configDigest=…} + ✅ logs containing a configDigest: Adding group / OCREndpoint Initialized / LatestConfig fetched + ✅ staging reports + telemetry actually arriving + ❌ ragep2p_* (one shared TCP conn) + ❌ rageping_* (per peer pair) + ❌ ragedisco_* and DiscoveryProtocol Status report (unions across groups) + ❌ ocr3_epoch & friends (Blue/Green collide on registration) + +THE TRAP + Identical node sets ⇒ staging peering is inherited from production and + every P2P dashboard is unchanged and green — whether or not staging works. + Verify adoption per node with ocr3_reporting_plugin_status, and verify + liveness with staging round/report progress. Nothing else proves it. + +THE OTHER TRAP + A peer ID that exists ONLY in the staging config cannot join: + bootstrap tracks the Blue slot, rejects it as an unknown peer, and never + relays its announcement. Add nodes via the slot the bootstrap tracks. +``` diff --git a/docs/llo-protocol-deep-dive.md b/docs/llo-protocol-deep-dive.md new file mode 100644 index 00000000..fb85ccdd --- /dev/null +++ b/docs/llo-protocol-deep-dive.md @@ -0,0 +1,552 @@ +# LLO Protocol Deep Dive: Rounds, Leaders, Outcomes, Reports & Gaps + +> A condensed reference for how the LLO (Data Streams) protocol behaves end-to-end: +> OCR3 round lifecycle, leader election & failure handling, the Observation→Outcome→Report +> pipeline, and what produces **report gaps** vs **large report ranges**. +> Includes signals for diagnosing healthy vs unhealthy DONs. + +--- + +## 1. The Two Layers + +The system is two cooperating layers. Confusing them is the #1 source of misunderstanding. + +| Layer | Owned by | Responsibility | State | +|---|---|---|---| +| **OCR3 protocol** | libocr | Leader election, rounds, epochs, consensus (Prepare/Commit), transmission scheduling | `SeqNr`, `Epoch`, `PreviousOutcome`, leader | +| **LLO plugin** | `llo/v30` | Application logic: what to observe, how to aggregate, when a channel is reportable, how to encode reports | `ValidAfterNanoseconds`, `ChannelDefinitions`, `StreamAggregates` (all carried *inside* the outcome) | + +**Critical invariant:** the LLO plugin is **stateless** w.r.t. the outcome chain. It receives `outctx.PreviousOutcome` from OCR3 and trusts that it is the last committed outcome (`SeqNr-1`). Failed rounds never reach the plugin. + +--- + +## 2. OCR3 Round Lifecycle (Observation → Outcome → Report) + +A *round* = one `SeqNr`. A round succeeds when it reaches **Certified Commit** (2f+1 commit signatures). Only then does `SeqNr` increment and the outcome become the new `PreviousOutcome`. + +### 2.1 Phases (leader side) + +```mermaid +flowchart TD + A[NewEpoch] --> B[SentEpochStart] + B --> C[SentRoundStart
broadcast MessageRoundStart + Query] + C --> D[Collect MessageObservation
from followers] + D --> E{ObservationQuorum?
2f+1 valid} + E -- no --> D + E -- yes --> F[Grace
wait DeltaGrace for stragglers] + F --> G[SentProposal
broadcast MessageProposal
with all signed observations] + G --> H[Collect MessagePrepare] + H --> I{2f+1 Prepare sigs?} + I -- no --> H + I -- yes --> J[CertifiedPrepare
persisted] + J --> K[Collect MessageCommit] + K --> L{2f+1 Commit sigs?} + L -- no --> K + L -- yes --> M[CertifiedCommit
SeqNr committed
outcome becomes PreviousOutcome] + M --> N[Reports generated
+ transmission scheduled] + N --> C +``` + +### 2.2 Phases (follower side) + +```mermaid +flowchart TD + A[NewEpoch] --> B[wait MessageEpochStart
from leader] + B -- timeout DeltaInitial --> Z[EventNewEpochRequest
trigger leader change] + B -- got msg --> C[NewRound] + C --> D[BackgroundObservation
call plugin.Observation] + D --> E[SentObservation
SendTo leader MessageObservation] + E --> F[wait MessageProposal
from leader] + F -- got msg --> G[BackgroundProposalOutcome
verify sigs, ValidateObservation,
ObservationQuorum, call plugin.Outcome] + G --> H[SentPrepare
broadcast MessagePrepare] + H --> I[wait 2f+1 Prepare sigs] + I --> J[SentCommit
broadcast MessageCommit] + J --> K[wait 2f+1 Commit sigs] + K --> L[CertifiedCommit
commit outcome] + L --> M[Reports generated] + M --> C +``` + +### 2.3 Where the LLO plugin hooks in + +| OCR3 phase | LLO plugin call | What it does | +|---|---|---| +| Round start (leader) | `Query()` | Returns `nil` (LLO doesn't need a query) | +| Observation (all nodes) | `Observation()` | Fetches stream values from `DataSource`, stamps `time.Now().UnixNano()` | +| Proposal (followers) | `ValidateObservation()` per obs | Checks observation well-formedness | +| Proposal (followers) | `ObservationQuorum()` | `2f+1` valid observations (default quorum) | +| Proposal (followers) | `Outcome()` | **The big one**: medianize timestamps, aggregate streams, update `ValidAfterNanoseconds`, carry channel defs | +| After commit | `Reports()` | For each reportable channel, encode a report | +| Transmission | `ShouldAcceptAttestedReport` / `ShouldTransmitAcceptedReport` | Both return `true` (transmit everything) | + +> **Performance note:** `Outcome()` runs on *every node* during the proposal phase (each follower computes it independently to verify the leader's proposal). It's pure and must be fast. The leader does **not** send the outcome in `MessageProposal` — it sends the signed observations, and each follower recomputes the outcome to check the digest matches. + +--- + +## 3. Leader Election & Failure Handling + +### 3.1 Leader selection is deterministic — no handshake + +```go +func Leader(epoch uint64, n int, key [16]byte) commontypes.OracleID { + // HMAC-based permutation of oracle IDs for this epoch + // Every node computes this locally. No message from the leader is needed. +} +``` + +All nodes independently compute `Leader(epoch, n, LeaderSelectionKey)`. **A dead node can be elected leader.** There is no "I accept leadership" confirmation. The protocol is *optimistic* — it assumes the leader is alive and relies on timeouts to detect otherwise. + +### 3.2 Happy path: leader is alive and fast + +```mermaid +sequenceDiagram + participant All as All Nodes + participant L as Leader (Node 5) + All->>All: Compute Leader(epoch=3) = Node 5 + Note over L: Sends MessageEpochStart (with 2f+1 EpochStartRequests) + L->>All: MessageEpochStart + loop each round + L->>All: MessageRoundStart + Query + All->>L: MessageObservation (signed) + L->>L: ObservationQuorum reached + L->>L: Wait DeltaGrace + L->>All: MessageProposal (all signed obs) + All->>All: Recompute Outcome, verify digest + All->>All: MessagePrepare + All->>All: MessageCommit + Note over All: CertifiedCommit → SeqNr++ → Reports + end + Note over All: DeltaProgress timer keeps resetting on each commit +``` + +### 3.2.1 Happy path: detailed per-phase message flow + +This expands one round into every OCR3 message and the LLO plugin call at each step. +Assumes `N=4, F=1` (so quorum = `2f+1 = 3`). + +```mermaid +sequenceDiagram + autonumber + participant L as Leader (Node 5) + participant F1 as Follower N1 + participant F2 as Follower N2 + participant F3 as Follower N3 + participant P as LLO Plugin + + Note over L,F3: ── EPOCH START ── + + L->>F1: MessageEpochStartRequest + L->>F2: MessageEpochStartRequest + L->>F3: MessageEpochStartRequest + Note over L: Collects 2f+1=3 EpochStartRequests
with valid HighestCertified proofs + L->>F1: MessageEpochStart (EpochStartProof + sig) + L->>F2: MessageEpochStart + L->>F3: MessageEpochStart + Note over F1,F3: Followers verify EpochStartProof
phase → NewRound + + Note over L,F3: ── ROUND (SeqNr=k) ── + + Note over L: phase → SentRoundStart
sets tRound = After(DeltaRound) + L->>F1: MessageRoundStart(epoch, seqNr, Query) + L->>F2: MessageRoundStart + L->>F3: MessageRoundStart + + Note over F1: phase → BackgroundObservation + F1->>P: Observation(ctx, outctx, query) + F2->>P: Observation(ctx, outctx, query) + F3->>P: Observation(ctx, outctx, query) + Note over P: DataSource.Observe() → stream values
stamp time.Now().UnixNano() + P-->>F1: encoded Observation + P-->>F2: encoded Observation + P-->>F3: encoded Observation + + Note over F1: phase → SentObservation + F1->>L: MessageObservation (SignedObservation) + F2->>L: MessageObservation + F3->>L: MessageObservation + + Note over L: Verify each SignedObservation sig
call plugin.ValidateObservation per obs + L->>P: ValidateObservation (per obs) + Note over L: ObservationQuorum? (2f+1=3 valid) + Note over L: YES → phase → Grace
set tGrace = After(DeltaGrace) + + Note over L: tGrace fires → phase → SentProposal + L->>F1: MessageProposal (AttributedSignedObservations) + L->>F2: MessageProposal + L->>F3: MessageProposal + Note over L: Leader does NOT send outcome,
only the signed observations + + Note over F1: phase → BackgroundProposalOutcome + F1->>P: ValidateObservation (per obs in proposal) + F2->>P: ValidateObservation + F3->>P: ValidateObservation + F1->>P: ObservationQuorum(aos) + F2->>P: ObservationQuorum(aos) + F3->>P: ObservationQuorum(aos) + Note over F1,F3: Each follower independently calls: + F1->>P: Outcome(ctx, outctx, query, aos) + F2->>P: Outcome(ctx, outctx, query, aos) + F3->>P: Outcome(ctx, outctx, query, aos) + Note over P: medianize timestamps
aggregate streams (median/quote)
update ValidAfterNanoseconds
carry ChannelDefinitions + P-->>F1: encoded Outcome + P-->>F2: encoded Outcome + P-->>F3: encoded Outcome + Note over F1,F3: Compute OutcomeInputsDigest + OutcomeDigest
Sign Prepare over (inputsDigest, outcomeDigest) + + Note over F1: phase → SentPrepare + F1->>L: MessagePrepare (PrepareSignature) + F1->>F2: MessagePrepare + F1->>F3: MessagePrepare + F2->>L: MessagePrepare + F2->>F1: MessagePrepare + F2->>F3: MessagePrepare + F3->>L: MessagePrepare + F3->>F1: MessagePrepare + F3->>F2: MessagePrepare + + Note over F1: Collect 2f+1=3 valid Prepare sigs
→ CertifiedPrepare (persisted) + Note over F1: phase → SentCommit
Sign Commit over outcomeDigest + F1->>L: MessageCommit (CommitSignature) + F1->>F2: MessageCommit + F1->>F3: MessageCommit + F2->>L: MessageCommit + F2->>F1: MessageCommit + F2->>F3: MessageCommit + F3->>L: MessageCommit + F3->>F1: MessageCommit + F3->>F2: MessageCommit + + Note over F1: Collect 2f+1=3 valid Commit sigs
→ CertifiedCommit + Note over L,F3: commit(outcome) → committedSeqNr = seqNr
outcome becomes PreviousOutcome for seqNr+1
SeqNr++ → next round + + Note over L,F3: ── REPORTS ── + + Note over L: Leader calls Reports(seqNr, outcome) + L->>P: Reports(ctx, seqNr, outcome) + Note over F1: Followers also call Reports + F1->>P: Reports(ctx, seqNr, outcome) + F2->>P: Reports(ctx, seqNr, outcome) + F3->>P: Reports(ctx, seqNr, outcome) + Note over P: ReportableChannels() → IsReportable per channel
for each reportable: encode report
emit ReportPlus[] (with ReportInfo) + P-->>L: []ReportPlus + P-->>F1: []ReportPlus + P-->>F2: []ReportPlus + P-->>F3: []ReportPlus + + Note over L,F3: ── TRANSMISSION ── + + Note over L,F3: ShouldAcceptAttestedReport → true
ShouldTransmitAcceptedReport → true + Note over L,F3: Per TransmissionSchedule:
oracles transmit in stages with delays
first successful transmit wins + L->>L: Transmit report (if scheduled) + F1->>F1: Transmit report (if scheduled) + Note over L,F3: LLO transmitter → Mercury server (gRPC)
or CRE transmitter → on-chain +``` + +**Key takeaways from the happy path:** + +1. **Epoch start is a one-time handshake** — leader proves it has 2f+1 `EpochStartRequest`s with valid `HighestCertified` proofs. Followers won't accept round messages until they've seen a valid `MessageEpochStart`. +2. **Observations go leader-bound only** (`SendTo`), not broadcast. The leader collects and re-distributes them in the proposal. +3. **The leader never sends the outcome** — it sends signed observations in `MessageProposal`. Every follower recomputes `Outcome()` independently and signs a `Prepare` over the resulting digest. This is how consensus on the outcome is reached *without* the leader dictating it. +4. **Prepare and Commit are broadcast** (all-to-all). The leader is just another node here — it also sends Prepare/Commit. +5. **`Outcome()` runs N times per round** (once per node), not once. This is the main CPU cost and why it must be pure and fast. +6. **Reports run on every node** after commit, but only scheduled transmitters actually send to Mercury/on-chain. +7. **`DeltaGrace` is the minimum round duration** under a correct leader — the leader always waits for stragglers after reaching quorum. + +### 3.3 Failure path A: leader is dead at epoch start + +```mermaid +sequenceDiagram + participant All as All Nodes + participant Dead as Leader (Node 5, DEAD) + All->>All: Compute Leader(epoch=3) = Node 5 + All->>All: Set tInitial = time.After(DeltaInitial) + Note over Dead: Never sends MessageEpochStart + All->>All: DeltaInitial fires → eventTInitialTimeout + All->>All: Send EventNewEpochRequest to Pacemaker + All->>All: Broadcast NewEpochWish(epoch=4) + Note over All: Once 2f+1 wish for epoch 4 → switch + All->>All: Compute Leader(epoch=4) = Node 2 (new leader) + Note over All: New epoch starts, Node 2 takes over +``` + +**Cost:** `DeltaInitial` of dead time. No rounds produced. Next successful round's report range spans this gap. + +### 3.4 Failure path B: leader starts, then can't complete rounds + +```mermaid +sequenceDiagram + participant All as All Nodes + participant L as Leader (Node 5, degrading) + All->>All: Leader(epoch=3) = Node 5 + L->>All: MessageEpochStart ✓ + L->>All: MessageRoundStart ✓ + All->>L: MessageObservation ✓ + Note over L: Network degrades / overloaded
can't gather 2f+1 / can't send proposal + All->>All: DeltaProgress fires (no commit in time) + All->>All: eventTProgressTimeout → EventNewEpochRequest + All->>All: Broadcast NewEpochWish(epoch=4) + Note over All: 2f+1 wishes → new epoch → new leader +``` + +**Cost:** `DeltaProgress` of dead time (can be multiple rounds worth if leader was partially working). The `RMax` cap also forces an epoch change after too many rounds even with a working leader (rotates leadership). + +### 3.5 Why no liveness pre-check? + +- A node can die *mid-epoch* — pre-checks can't prevent that +- Pre-checks add latency on the happy path +- Timeouts handle all failure modes uniformly (dead, slow, partitioned, Byzantine) +- Deterministic `Leader()` avoids a meta-election problem + +--- + +## 4. The Outcome: Where Report Ranges Are Born + +### 4.1 Outcome structure + +``` +Outcome { + LifeCycleStage // staging | production | retired + ObservationTimestampNanoseconds // median of all observation timestamps + ChannelDefinitions // set of channels + ValidAfterNanoseconds // per-channel: "report covers [ValidAfter, ObsTs]" + StreamAggregates // per-stream/aggregator: medianized values +} +``` + +### 4.2 How `ValidAfterNanoseconds` advances + +This is the **single most important mechanism** for understanding gaps vs ranges. + +```mermaid +flowchart TD + A[Previous Outcome] --> B{Was channel reportable
in previous outcome?} + B -- yes --> C[ValidAfter ← previousOutcome.ObservationTimestampNanoseconds
ADVANCES] + B -- no --> D[ValidAtter ← previous ValidAfter
STAYS SAME] + C --> E[New report covers
prevObsTs → currentObsTs] + D --> F[New report covers
oldValidAfter → currentObsTs
EXTENDED RANGE] +``` + +**Reportable** = passes `IsReportable()`: +- Not retired, not tombstoned +- Has `ValidAfterNanoseconds` entry +- `obsTs >= validAfter + minReportInterval` (timing) +- For seconds-resolution: `validAfterSeconds < obsTsSeconds` (no overlap) +- If `DisableNilStreamValues=true`: all stream aggregates present (non-nil) + +### 4.3 The golden rule + +> **`ValidAfter` only advances when the previous outcome's channel was reportable.** If it wasn't reportable, `ValidAfter` stays put, and the next report covers a *longer* range. This is by design — it prevents gaps. + +--- + +## 5. Gaps vs Large Ranges + +### 5.1 Definitions + +- **Report range** = `[ValidAfterNanoseconds, ObservationTimestampNanoseconds]` +- **Large range** = a single report covering a long time span (e.g. 30s). *Contiguous, no missing data.* +- **Gap** = a time range with *no report at all*. E.g. reports `[1,2], [3,3], [4,5]` then `[7,7]` — `[6,6]` is missing. + +### 5.2 What causes LARGE RANGES (not gaps) + +All of these skip rounds entirely. `ValidAfter` doesn't move. Next success covers the whole gap. + +| Cause | Mechanism | Cost | +|---|---|---| +| Failed round (no consensus) | `SeqNr` doesn't advance, no outcome | Time until next successful round | +| Dead leader (epoch start) | `DeltaInitial` timeout → epoch change | `DeltaInitial` | +| Slow/degraded leader | `DeltaProgress` timeout → epoch change | `DeltaProgress` | +| Leader rotation (`RMax`) | Forced epoch change after N rounds | One epoch transition | +| Slow `DataSource.Observe` | Observation timestamp captured late | Round takes longer | +| Network latency to leader | Observations arrive slowly | Round takes longer | +| DON-wide overload | All nodes slow | Consistently larger ranges | + +### 5.3 What causes actual GAPS + +A gap requires **two conditions simultaneously**: +1. Channel **passes `IsReportable`** → `ValidAfter` advances to `previousObsTs` +2. Report is **silently dropped at encode** → no report covers `[prevObsTs, currentObsTs]` + +```mermaid +flowchart TD + A[Round N: channel reportable?] -- yes --> B[ValidAfter advances to obsTs_N] + B --> C[Encode report] + C -- success --> D[Report covers range ✓] + C -- FAILS --> E[No report produced
but ValidAfter already advanced] + E --> F[Round N+1: report covers obsTs_N → obsTs_N+1] + F --> G[GAP: obsTs_N-1 → obsTs_N uncovered!] +``` + +**Encode-drop failure modes:** +- `DisableNilStreamValues=false` + nil stream value → `encodeReport` fails on `ErrNilStreamValue` +- Report codec error (e.g. bid/mid/ask validation failure in EVM codecs) +- Missing codec for report format + +> **`DisableNilStreamValues=true` prevents gaps from nil stream values** by making the channel unreportable at `IsReportable` time (so `ValidAfter` doesn't advance). But it does **not** prevent gaps from codec/validation failures at encode time. + +### 5.4 Quick reference table + +| Scenario | `IsReportable` | Report produced? | `ValidAfter` | Result | +|---|---|---|---|---| +| Healthy round | ✓ | ✓ | Advances | Normal range | +| Nil stream, `DisableNilStreamValues=true` | ✗ (blocked) | ✗ | Stays | **Extended range** (no gap) | +| Nil stream, `DisableNilStreamValues=false` | ✓ | ✗ (encode fails) | Advances | **GAP** | +| Codec/validation error at encode | ✓ | ✗ (encode fails) | Advances | **GAP** | +| Failed round (no consensus) | n/a | n/a | n/a (no outcome) | **Large range** on next success | +| Dead leader | n/a | n/a | n/a | **Large range** on next success | + +--- + +## 6. LLO-Specific: When Medianization Fails + +A channel needs stream values from `StreamAggregates`. Aggregation fails when fewer than `f+1` observations have a usable value for that stream. + +```go +// MedianAggregator +if len(observations) <= f { + return nil, fmt.Errorf("not enough observations to calculate median, expected at least f+1, got %d", len(observations)) +} +``` + +When aggregation fails: +- The stream ID is **missing** from `StreamAggregates` (not nil, just absent) +- If `DisableNilStreamValues=true`: channel fails `IsReportable` → `ValidAfter` doesn't advance → **extended range, no gap** +- If `DisableNilStreamValues=false`: channel may pass `IsReportable` but `encodeReport` fails on nil → **gap** + +For `TimestampedStreamValue`, failed aggregation **carries forward** the previous value (monotonicity guarantee) — so the channel may still be reportable with a stale value. + +--- + +## 7. Protocol Startup & Outcome Chain + +```mermaid +sequenceDiagram + participant OCR3 + participant Plugin + Note over OCR3: SeqNr=1, PreviousOutcome=nil + OCR3->>Plugin: Outcome(SeqNr=1, ...) + Note over Plugin: Special case: returns cornerstone outcome
{stage, 0, nil, nil, nil} + Plugin->>OCR3: Cornerstone outcome + Note over OCR3: SeqNr=2, PreviousOutcome=cornerstone + OCR3->>Plugin: Outcome(SeqNr=2, cornerstone, observations) + Note over Plugin: ValidAfterNanoseconds nil → new channels get
ValidAfter = currentObsTs + Plugin->>OCR3: Outcome with channel defs + ValidAfter + Note over OCR3: SeqNr=3, PreviousOutcome=above + OCR3->>Plugin: Outcome(SeqNr=3, ...) + Note over Plugin: Channels now have ValidAfter entries
Normal reporting begins +``` + +**Key points:** +- `SeqNr=1` always returns the empty cornerstone outcome (early return, ignores `PreviousOutcome`) +- `SeqNr=2` is where channels get initialized: `ValidAfter = outcome.ObservationTimestampNanoseconds` +- `SeqNr=3+` is normal operation +- Failed rounds don't increment `SeqNr` — the chain is unbroken + +--- + +## 8. DON Health Signals + +### 8.1 Healthy DON signals + +| Signal | Where to look | Healthy value | +|---|---|---| +| Report cadence | Report telemetry / on-chain | Consistent, matches `minReportInterval` / seconds resolution | +| Report range span | `obsTs - validAfter` per report | ~1s for seconds-resolution (minimum); stable | +| Epoch churn | libocr logs `EpochStarted`, metrics `ocr3_epoch` | Low; epochs last many rounds | +| Round success rate | libocr metrics | High; few `TProgress`/`TInitial` timeouts | +| `SeqNr` increments | Outcome telemetry | Monotonic, regular | +| Observation count per round | Outcome telemetry | `2f+1` or close to `N` | +| Transmit queue depth | `llo_mercurytransmitter_transmit_queue_load` | Low, not growing | +| Consecutive transmit errors | `llo_mercurytransmitter_concurrent_transmit_gauge` | 0 | + +### 8.2 Unhealthy DON signals + +| Signal | Likely cause | Impact | +|---|---|---| +| Report range span suddenly jumps to `~DeltaProgress` | Dead/slow leader → epoch change | Large range (not a gap) | +| Report range span jumps to `~DeltaInitial` | Leader dead at epoch start | Large range | +| Epoch changing every few rounds | Leader keeps failing / `RMax` too low / network partition | Frequent large ranges | +| `TProgress fired` logs frequent | Leader can't complete rounds | Large ranges | +| `TInitial fired` logs frequent | Leaders dying at start | Large ranges | +| `ObservationQuorum returned false despite n-f` logs | Plugin bug or widespread observation failures | Rounds fail → large ranges | +| Gaps in report `ValidAfter` sequence | Encode-time drops (codec errors, nil values with `DisableNilStreamValues=false`) | **Actual gaps** | +| `dropping MessageObservation carrying invalid` logs | Byzantine/misbehaving node | Reduced observation count | +| One node's observations consistently dropped | That node is faulty (bad data, bad sigs, slow) | Identify & remove | +| Transmit queue > 50% full | Mercury server unreachable / slow | Reports delayed (not gaps in protocol, but delivery lag) | +| `concurrent_transmit_gauge` at max | Transmit threads saturated | Delivery bottleneck | + +### 8.3 Diagnosing a bad node + +1. **Check if a specific node is never leader but others rotate normally** → node may be partitioned or down (not receiving `NewEpochWish` messages) +2. **Check if a specific node's observations are always dropped** (`dropping MessageObservation` logs from leader) → node sending invalid/garbage data +3. **Check if epochs die when a specific node is leader** → that node can't lead (overloaded, slow disk, bad network) +4. **Check `includedObservationsTotal` metric per node** → a node consistently not included in proposals is being excluded by leaders +5. **Check report telemetry `StreamValues`** → if one node's values are consistently outliers, it may have a bad data source + +### 8.4 Diagnosing gaps vs large ranges + +``` +Look at sequence of (ValidAfter, ObservationTimestamp) pairs: + +Healthy: [0,1] [1,2] [2,3] [3,4] ← each range contiguous with next +Large range: [0,1] [1,30] [30,31] ← [1,30] spans a failure period, but NO gap +GAP: [0,1] [1,2] [4,5] ← [2,4] missing entirely = GAP +``` + +- **Large range**: `ValidAfter_N == ObservationTimestamp_N-1` (contiguous) but span is large → round/epoch failure, expected behavior +- **Gap**: `ValidAfter_N > ObservationTimestamp_N-1` → encode-time drop, investigate codec errors / nil stream values + +--- + +## 9. Config Parameters That Matter + +| Parameter | Effect | Tuning notes | +|---|---|---| +| `DeltaRound` | Min time between round starts | Lower bound only; actual rounds may take longer | +| `DeltaGrace` | Leader waits for stragglers after quorum | Rounds always take ≥ `DeltaGrace` under correct leaders | +| `DeltaProgress` | Max time without a commit before epoch change | Too short → premature epoch churn (liveness failure) | +| `DeltaInitial` | Max time without `MessageEpochStart` before epoch change | Too short → premature epoch churn | +| `RMax` | Max rounds per epoch | Forces leader rotation; too low → unnecessary churn | +| `DefaultMinReportIntervalNanoseconds` | Min time between reports per channel | Enforced in `IsReportable` | +| `DisableNilStreamValues` (per channel) | Blocks reportability on nil stream values | `true` = gap prevention for nil values | + +--- + +## 10. Summary Cheat Sheet + +``` +FAILED ROUND / DEAD LEADER / SLOW DON + → SeqNr doesn't advance + → PreviousOutcome carries forward unchanged + → Next success: ValidAfter unchanged, obsTs = now + → ONE BIG RANGE (contiguous, no gap) + +NIL STREAM VALUE + DisableNilStreamValues=true + → IsReportable fails + → ValidAfter doesn't advance + → EXTENDED RANGE (no gap) + +NIL STREAM VALUE + DisableNilStreamValues=false + → IsReportable passes + → ValidAfter advances + → encodeReport fails on nil + → GAP + +CODEC/VALIDATION ERROR AT ENCODE + → IsReportable passes (all values present) + → ValidAfter advances + → encodeReport fails + → GAP + +HEALTHY DON + → Consistent report cadence + → Range spans ~minReportInterval (or 1s for seconds-res) + → Low epoch churn + → 2f+1 observations per round + +UNHEALTHY DON + → Range span jumps ≈ DeltaProgress/DeltaInitial + → Frequent epoch changes + → TProgress/TInitial timeouts in logs + → Gaps = encode drops (check codec errors, nil values) +``` diff --git a/llo/libocr b/llo/libocr new file mode 160000 index 00000000..0cb21f3a --- /dev/null +++ b/llo/libocr @@ -0,0 +1 @@ +Subproject commit 0cb21f3a2b0b3b607e5d899bdbba332c04a9e146 diff --git a/llo/plugin_outcome_test.go b/llo/plugin_outcome_test.go new file mode 100644 index 00000000..a8e52954 --- /dev/null +++ b/llo/plugin_outcome_test.go @@ -0,0 +1,1053 @@ +package llo + +import ( + "fmt" + "os" + "testing" + "time" + + "github.com/shopspring/decimal" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/smartcontractkit/libocr/commontypes" + "github.com/smartcontractkit/libocr/offchainreporting2/types" + "github.com/smartcontractkit/libocr/offchainreporting2plus/ocr3types" + + "github.com/smartcontractkit/chainlink-common/pkg/logger" + llotypes "github.com/smartcontractkit/chainlink-common/pkg/types/llo" + "github.com/smartcontractkit/chainlink-common/pkg/utils/tests" +) + +func Test_Outcome(t *testing.T) { + for _, codec := range []OutcomeCodec{protoOutcomeCodecV0{}, protoOutcomeCodecV1{}} { + t.Run(fmt.Sprintf("OutcomeCodec: %T", codec), func(t *testing.T) { + testOutcome(t, codec) + }) + } +} + +// Test_Outcome_GoldenFiles verifies that Plugin.Outcome() correctly decodes and advances from +// golden-encoded previous outcomes. Uses V1 codec only; golden files live in testdata/outcome_serialization/. +func Test_Outcome_GoldenFiles(t *testing.T) { + ctx := tests.Context(t) + obsCodec, err := NewProtoObservationCodec(logger.Nop(), true) + require.NoError(t, err) + codec := OffchainConfig{ + ProtocolVersion: 1, + DefaultMinReportIntervalNanoseconds: 1, + }.GetOutcomeCodec() + p := &Plugin{ + Config: Config{true}, + OutcomeCodec: codec, + Logger: logger.Test(t), + ObservationCodec: obsCodec, + DonID: 10000043, + ConfigDigest: types.ConfigDigest{1, 2, 3, 4}, + ProtocolVersion: 1, + DefaultMinReportIntervalNanoseconds: 1, + } + // Minimal observations (timestamp only) so the plugin advances from previous outcome without new channel defs or stream values. + obs, err := p.ObservationCodec.Encode(Observation{UnixTimestampNanoseconds: 9876543210 + uint64(time.Second)}) + require.NoError(t, err) + aos := make([]types.AttributedObservation, 4) + for i := range aos { + aos[i] = types.AttributedObservation{Observation: obs, Observer: commontypes.OracleID(i)} + } + + for _, tc := range GoldenOutcomeCases() { + t.Run(tc.Name, func(t *testing.T) { + golden, err := os.ReadFile(tc.OutputFile) + if err != nil { + if os.IsNotExist(err) { + t.Skip("golden file not found; run llo/tools/generate_golden to generate") + } + require.NoError(t, err) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{ + PreviousOutcome: golden, + SeqNr: 2, + }, types.Query{}, aos) + require.NoError(t, err) + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + // Plugin should have decoded the golden previous outcome and produced a valid next outcome. + assert.Equal(t, tc.Outcome.LifeCycleStage, decoded.LifeCycleStage) + if len(tc.Outcome.ChannelDefinitions) > 0 { + assert.Equal(t, tc.Outcome.ChannelDefinitions, decoded.ChannelDefinitions) + } + }) + } +} + +// Test_Outcome_EncodedMatchesGolden verifies that Plugin.Outcome() encoded output matches golden files +// for scenarios that the plugin can produce. Runs only for "initial" (seq 1) and "from_full" (previous=full). +func Test_Outcome_EncodedMatchesGolden(t *testing.T) { + ctx := tests.Context(t) + obsCodec, err := NewProtoObservationCodec(logger.Nop(), true) + require.NoError(t, err) + codec := OffchainConfig{ + ProtocolVersion: 1, + DefaultMinReportIntervalNanoseconds: 1, + }.GetOutcomeCodec() + p := &Plugin{ + Config: Config{true}, + OutcomeCodec: codec, + Logger: logger.Test(t), + ObservationCodec: obsCodec, + DonID: 10000043, + ConfigDigest: types.ConfigDigest{1, 2, 3, 4}, + ProtocolVersion: 1, + DefaultMinReportIntervalNanoseconds: 1, + } + + // Golden cases that the plugin produces; "full" is only used as previous outcome, not produced here. + pluginProducedCases := map[string]bool{"initial": true, "from_full": true} + + for _, tc := range GoldenOutcomeCases() { + if !pluginProducedCases[tc.Name] { + continue + } + t.Run(tc.Name, func(t *testing.T) { + golden, err := os.ReadFile(tc.OutputFile) + if err != nil { + if os.IsNotExist(err) { + t.Skip("golden file not found; run llo/tools/generate_golden to generate") + } + require.NoError(t, err) + } + + var outcome ocr3types.Outcome + switch tc.Name { + case "initial": + emptyObs, err := p.ObservationCodec.Encode(Observation{}) + require.NoError(t, err) + aos := make([]types.AttributedObservation, 4) + for i := range aos { + aos[i] = types.AttributedObservation{Observation: emptyObs, Observer: commontypes.OracleID(i)} + } + outcome, err = p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 1}, types.Query{}, aos) + require.NoError(t, err) + case "from_full": + fullGolden, err := os.ReadFile("testdata/outcome_serialization/full.bin") + require.NoError(t, err) + obsTS := uint64(9876543210 + 1e9) + obs, err := p.ObservationCodec.Encode(Observation{UnixTimestampNanoseconds: obsTS}) + require.NoError(t, err) + aos := make([]types.AttributedObservation, 4) + for i := range aos { + aos[i] = types.AttributedObservation{Observation: obs, Observer: commontypes.OracleID(i)} + } + outcome, err = p.Outcome(ctx, ocr3types.OutcomeContext{ + PreviousOutcome: fullGolden, + SeqNr: 2, + }, types.Query{}, aos) + require.NoError(t, err) + default: + t.Skipf("plugin does not produce golden case %q", tc.Name) + return + } + require.NoError(t, err) + assert.Equal(t, golden, []byte(outcome), "Plugin.Outcome() encoded output should match golden file") + }) + } +} + +func testOutcome(t *testing.T, outcomeCodec OutcomeCodec) { + ctx := tests.Context(t) + + obsCodec, err := NewProtoObservationCodec(logger.Nop(), true) + require.NoError(t, err) + p := &Plugin{ + Config: Config{true}, + OutcomeCodec: outcomeCodec, + Logger: logger.Test(t), + ObservationCodec: obsCodec, + DonID: 10000043, + ConfigDigest: types.ConfigDigest{1, 2, 3, 4}, + } + testStartTS := time.Now() + testStartNanos := uint64(testStartTS.UnixNano()) //nolint:gosec // safe cast in tests + + t.Run("if number of observers < 2f+1, errors", func(t *testing.T) { + _, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 1}, types.Query{}, []types.AttributedObservation{}) + require.EqualError(t, err, "invariant violation: expected at least 2f+1 attributed observations, got 0 (f: 0)") + p.F = 1 + _, err = p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 1}, types.Query{}, []types.AttributedObservation{{}, {}}) + require.EqualError(t, err, "invariant violation: expected at least 2f+1 attributed observations, got 2 (f: 1)") + }) + + t.Run("if seqnr == 1, and has enough observers, emits initial outcome with 'production' LifeCycleStage", func(t *testing.T) { + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 1}, types.Query{}, []types.AttributedObservation{ + { + Observation: []byte{}, + Observer: commontypes.OracleID(0), + }, + { + Observation: []byte{}, + Observer: commontypes.OracleID(1), + }, + { + Observation: []byte{}, + Observer: commontypes.OracleID(2), + }, + { + Observation: []byte{}, + Observer: commontypes.OracleID(3), + }, + }) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, Outcome{ + LifeCycleStage: "production", + }, decoded) + }) + + t.Run("channel definitions", func(t *testing.T) { + t.Run("adds a new channel definition if there are enough votes", func(t *testing.T) { + newCd := llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormat(2), + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorMedian}}, + } + obs, err := p.ObservationCodec.Encode(Observation{ + UpdateChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 42: newCd, + }, + }) + require.NoError(t, err) + aos := []types.AttributedObservation{} + for i := uint8(0); i < 4; i++ { + aos = append(aos, + types.AttributedObservation{ + Observation: obs, + Observer: commontypes.OracleID(i), + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, newCd, decoded.ChannelDefinitions[42]) + }) + + t.Run("replaces an existing channel definition if there are enough votes", func(t *testing.T) { + newCd := llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormat(2), + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorQuote}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorMedian}}, + } + obs, err := p.ObservationCodec.Encode(Observation{ + UpdateChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 42: newCd, + }, + }) + require.NoError(t, err) + aos := []types.AttributedObservation{} + for i := uint8(0); i < 4; i++ { + aos = append(aos, + types.AttributedObservation{ + Observation: obs, + Observer: commontypes.OracleID(i), + }) + } + + previousOutcome, err := p.OutcomeCodec.Encode(Outcome{ + ChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 42: { + ReportFormat: llotypes.ReportFormat(1), + Streams: []llotypes.Stream{{StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorMedian}, {StreamID: 4, Aggregator: llotypes.AggregatorMedian}}, + }, + }, + }) + require.NoError(t, err) + + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{PreviousOutcome: previousOutcome, SeqNr: 2}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, newCd, decoded.ChannelDefinitions[42]) + }) + + t.Run("replaces channel definition with tombstoned version and stops generating reports", func(t *testing.T) { + channelID := llotypes.ChannelID(42) + originalCd := llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormatJSON, + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}}, + Tombstone: false, + } + tombstonedCd := llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormatJSON, + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}}, + Tombstone: true, + } + + // Create previous outcome with a non-tombstoned, reportable channel + previousObsTS := testStartNanos + uint64(2*time.Second) + previousOutcome := Outcome{ + LifeCycleStage: LifeCycleStageProduction, + ObservationTimestampNanoseconds: previousObsTS, + ChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + channelID: originalCd, + }, + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + channelID: testStartNanos, // Channel is reportable + }, + } + + // Verify channel is reportable before tombstoning + require.Nil(t, previousOutcome.IsReportable(channelID, 1, uint64(100*time.Millisecond))) + reportable, _ := previousOutcome.ReportableChannels(1, uint64(100*time.Millisecond)) + assert.Contains(t, reportable, channelID) + + // Encode previous outcome + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + + // Create observations voting to update channel to tombstoned version + obs, err := p.ObservationCodec.Encode(Observation{ + UpdateChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + channelID: tombstonedCd, + }, + UnixTimestampNanoseconds: previousObsTS + uint64(1*time.Second), + }) + require.NoError(t, err) + + aos := []types.AttributedObservation{} + for i := uint8(0); i < 4; i++ { + aos = append(aos, + types.AttributedObservation{ + Observation: obs, + Observer: commontypes.OracleID(i), + }) + } + + // Generate new outcome with tombstoned channel + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{ + PreviousOutcome: encodedPreviousOutcome, + SeqNr: 3, + }, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + // Verify channel definition was replaced with tombstoned version + assert.True(t, decoded.ChannelDefinitions[channelID].Tombstone, "Channel should be tombstoned") + assert.Equal(t, tombstonedCd, decoded.ChannelDefinitions[channelID]) + + // Verify channel is no longer reportable + err = decoded.IsReportable(channelID, 1, uint64(100*time.Millisecond)) + require.NotNil(t, err) + assert.Contains(t, err.Error(), "tombstone channel") + + // Verify ReportableChannels excludes the tombstoned channel + reportable, unreportable := decoded.ReportableChannels(1, uint64(100*time.Millisecond)) + assert.NotContains(t, reportable, channelID, "Tombstoned channel should not be in reportable list") + require.Len(t, unreportable, 1) + assert.Equal(t, channelID, unreportable[0].ChannelID) + assert.Contains(t, unreportable[0].Error(), "tombstone channel") + }) + + t.Run("does not add channels beyond MaxOutcomeChannelDefinitionsLength", func(t *testing.T) { + newCd := llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormat(2), + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorMedian}}, + } + obs := Observation{UpdateChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{}} + for i := uint32(0); i < MaxOutcomeChannelDefinitionsLength+10; i++ { + obs.UpdateChannelDefinitions[i] = newCd + } + encoded, err := p.ObservationCodec.Encode(obs) + require.NoError(t, err) + aos := []types.AttributedObservation{} + for i := uint8(0); i < 4; i++ { + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Len(t, decoded.ChannelDefinitions, MaxOutcomeChannelDefinitionsLength) + + // should contain channels 0 thru 999 + assert.Contains(t, decoded.ChannelDefinitions, llotypes.ChannelID(0)) + assert.Contains(t, decoded.ChannelDefinitions, llotypes.ChannelID(MaxOutcomeChannelDefinitionsLength-1)) + assert.NotContains(t, decoded.ChannelDefinitions, llotypes.ChannelID(MaxOutcomeChannelDefinitionsLength)) + assert.NotContains(t, decoded.ChannelDefinitions, llotypes.ChannelID(MaxOutcomeChannelDefinitionsLength+1)) + }) + }) + + t.Run("stream observations", func(t *testing.T) { + smallDefinitions := map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 1: { + ReportFormat: llotypes.ReportFormatJSON, + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorQuote}}, + }, + 2: { + ReportFormat: llotypes.ReportFormatEVMPremiumLegacy, + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorQuote}}, + }, + } + + t.Run("aggregates values when all stream values are present from all observers", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: testStartNanos, + ChannelDefinitions: smallDefinitions, + ValidAfterNanoseconds: nil, + StreamAggregates: nil, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + outctx := ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome} + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: testStartNanos + uint64(time.Second) + uint64(i*100)*uint64(time.Millisecond), //nolint:gosec // safe cast in tests + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: ToDecimal(decimal.NewFromInt(int64(100 + i*10))), + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(200 + i*10)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(300 + i*10)), Benchmark: decimal.NewFromInt(int64(310 + i*10)), Ask: decimal.NewFromInt(int64(320 + i*10))}, + }} + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, outctx, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + observationsTs := decoded.ObservationTimestampNanoseconds + assert.GreaterOrEqual(t, observationsTs, uint64(testStartTS.UnixNano()+1_200_000_000)) //nolint:gosec // time won't be negative + + // NOTE: In protoOutcomeCodecV0 precision is lost on timestamp + // serialization, so validAfterNanoseconds will be truncated to + // seconds + expectedValidAfterSeconds := observationsTs + if _, ok := p.OutcomeCodec.(protoOutcomeCodecV0); ok { + expectedValidAfterSeconds = (observationsTs / 1e9) * 1e9 + } + + assert.Equal(t, Outcome{ + LifeCycleStage: "test", + ObservationTimestampNanoseconds: observationsTs, + ChannelDefinitions: smallDefinitions, + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: expectedValidAfterSeconds, // set to median observation timestamp + 2: expectedValidAfterSeconds, + }, + StreamAggregates: map[llotypes.StreamID]map[llotypes.Aggregator]StreamValue{ + 1: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(120)), + }, + 2: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(220))}, + }, + 3: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorQuote: &Quote{Bid: decimal.NewFromInt(320), Benchmark: decimal.NewFromInt(330), Ask: decimal.NewFromInt(340)}, + }, + }, + }, decoded) + }) + t.Run("unreportable channels from the previous outcome re-use the same previous ValidAfterNanoseconds", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: uint64(102030410 * time.Second), + ChannelDefinitions: nil, // nil channel definitions makes all channels unreportable + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: uint64(102030405 * time.Second), + 2: uint64(102030400 * time.Second), + }, + StreamAggregates: map[llotypes.StreamID]map[llotypes.Aggregator]StreamValue{ + 1: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(120)), + }, + 2: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(220)), + }, + 3: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorQuote: &Quote{Bid: decimal.NewFromInt(320), Benchmark: decimal.NewFromInt(330), Ask: decimal.NewFromInt(340)}, + }, + }, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: uint64(102030415 * time.Second), + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: ToDecimal(decimal.NewFromInt(int64(120))), + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(220)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(320)), Benchmark: decimal.NewFromInt(int64(330)), Ask: decimal.NewFromInt(int64(340))}, + }, + } + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, uint64(102030415*time.Second), decoded.ObservationTimestampNanoseconds) + require.Len(t, decoded.ValidAfterNanoseconds, 2) + assert.Equal(t, uint64(102030405*time.Second), decoded.ValidAfterNanoseconds[1]) + assert.Equal(t, uint64(102030400*time.Second), decoded.ValidAfterNanoseconds[2]) + }) + t.Run("ValidAfterNanoseconds is set based on the previous observation timestamp such that reports never overlap", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: uint64(102030410 * time.Second), + ChannelDefinitions: smallDefinitions, + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: uint64(102030405 * time.Second), + 2: uint64(102030400 * time.Second), + }, + StreamAggregates: map[llotypes.StreamID]map[llotypes.Aggregator]StreamValue{ + 1: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(120)), + }, + 2: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(220)), + }, + 3: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorQuote: &Quote{Bid: decimal.NewFromInt(320), Benchmark: decimal.NewFromInt(330), Ask: decimal.NewFromInt(340)}, + }, + }, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: uint64(102030415 * time.Second), + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: ToDecimal(decimal.NewFromInt(int64(120))), + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(220)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(320)), Benchmark: decimal.NewFromInt(int64(330)), Ask: decimal.NewFromInt(int64(340))}, + }, + } + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, uint64(102030415*time.Second), decoded.ObservationTimestampNanoseconds) + require.Len(t, decoded.ValidAfterNanoseconds, 2) + assert.Equal(t, uint64(102030410*time.Second), decoded.ValidAfterNanoseconds[1]) + assert.Equal(t, uint64(102030410*time.Second), decoded.ValidAfterNanoseconds[2]) + }) + t.Run("does generate outcome for reports that would overlap on a seconds-basis (allows duplicate reports)", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: uint64(102030410 * time.Second), + ChannelDefinitions: smallDefinitions, + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: uint64(102030409 * time.Second), + 2: uint64(102030409 * time.Second), + }, + StreamAggregates: map[llotypes.StreamID]map[llotypes.Aggregator]StreamValue{ + 1: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(120)), + }, + 2: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: ToDecimal(decimal.NewFromInt(220)), + }, + 3: map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorQuote: &Quote{Bid: decimal.NewFromInt(320), Benchmark: decimal.NewFromInt(330), Ask: decimal.NewFromInt(340)}, + }, + }, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: uint64((102030410 * time.Second) + 100*time.Millisecond), // 100ms after previous outcome + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: ToDecimal(decimal.NewFromInt(int64(120))), + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(220)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(320)), Benchmark: decimal.NewFromInt(int64(330)), Ask: decimal.NewFromInt(int64(340))}, + }, + } + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, uint64(102030410*time.Second+100*time.Millisecond), decoded.ObservationTimestampNanoseconds) + require.Len(t, decoded.ValidAfterNanoseconds, 2) + assert.Equal(t, uint64(102030410*time.Second), decoded.ValidAfterNanoseconds[1]) + assert.Equal(t, uint64(102030410*time.Second), decoded.ValidAfterNanoseconds[2]) + }) + t.Run("aggregation function returns error", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: testStartNanos, + ChannelDefinitions: smallDefinitions, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + outctx := ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome} + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + var sv StreamValue + // only one reported a value; not enough + if i == 0 { + sv = ToDecimal(decimal.NewFromInt(100)) + } + obs := Observation{ + UnixTimestampNanoseconds: testStartNanos + uint64(time.Second) + uint64(i*100)*uint64(time.Millisecond), //nolint:gosec // safe cast in tests + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: sv, + // 2 and 3 ok + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(220)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(320)), Benchmark: decimal.NewFromInt(int64(330)), Ask: decimal.NewFromInt(int64(340))}, + }} + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, outctx, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + // NOTE: `1` is missing because of insufficient observations + assert.Len(t, decoded.StreamAggregates, 2) + assert.Contains(t, decoded.StreamAggregates, llotypes.StreamID(2)) + assert.Contains(t, decoded.StreamAggregates, llotypes.StreamID(3)) + assert.Equal(t, map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorMedian: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(220))}, + }, decoded.StreamAggregates[2]) + assert.Equal(t, map[llotypes.Aggregator]StreamValue{ + llotypes.AggregatorQuote: &Quote{Bid: decimal.NewFromInt(320), Benchmark: decimal.NewFromInt(330), Ask: decimal.NewFromInt(340)}, + }, decoded.StreamAggregates[3]) + }) + t.Run("sends outcome telemetry if channel is specified", func(t *testing.T) { + ch := make(chan *LLOOutcomeTelemetry, 10000) + p.OutcomeTelemetryCh = ch + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: testStartNanos, + ChannelDefinitions: smallDefinitions, + ValidAfterNanoseconds: nil, + StreamAggregates: nil, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + outctx := ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome} + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: testStartNanos + uint64(time.Second) + uint64(i*100)*uint64(time.Millisecond), //nolint:gosec // safe cast in tests + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: ToDecimal(decimal.NewFromInt(int64(100 + i*10))), + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 123456789, StreamValue: ToDecimal(decimal.NewFromInt(int64(200 + i*10)))}, + 3: &Quote{Bid: decimal.NewFromInt(int64(300 + i*10)), Benchmark: decimal.NewFromInt(int64(310 + i*10)), Ask: decimal.NewFromInt(int64(320 + i*10))}, + }} + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, outctx, types.Query{}, aos) + require.NoError(t, err) + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + telem := <-ch + assert.Equal(t, string(decoded.LifeCycleStage), telem.LifeCycleStage) + assert.Equal(t, decoded.ObservationTimestampNanoseconds, telem.ObservationTimestampNanoseconds) + assert.Equal(t, len(decoded.ChannelDefinitions), len(telem.ChannelDefinitions)) + assert.Equal(t, len(decoded.ValidAfterNanoseconds), len(telem.ValidAfterNanoseconds)) + assert.Equal(t, len(decoded.StreamAggregates), len(telem.StreamAggregates)) + assert.Equal(t, uint64(2), telem.SeqNr) + assert.Equal(t, p.ConfigDigest[:], telem.ConfigDigest) + assert.Equal(t, p.DonID, telem.DonId) + }) + t.Run("handles TimestampedStreamValue correctly", func(t *testing.T) { + timestamped := map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 1: { + ReportFormat: llotypes.ReportFormatJSON, + Streams: []llotypes.Stream{ + {StreamID: 1, Aggregator: llotypes.AggregatorMedian}, + {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, + {StreamID: 3, Aggregator: llotypes.AggregatorMedian}, + }, + }, + } + t.Run("writes values in if its a brand new stream", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("test"), + ObservationTimestampNanoseconds: testStartNanos, + ChannelDefinitions: timestamped, + ValidAfterNanoseconds: nil, + StreamAggregates: nil, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + outctx := ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome} + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: testStartNanos + uint64(time.Second) + uint64(i*100)*uint64(time.Millisecond), //nolint:gosec // safe cast in tests + StreamValues: map[llotypes.StreamID]StreamValue{ + 1: &TimestampedStreamValue{ObservedAtNanoseconds: 100000000 + uint64(i), StreamValue: ToDecimal(decimal.NewFromInt(int64(100 + i)))}, //nolint:gosec // will never be > 4 + 2: &TimestampedStreamValue{ObservedAtNanoseconds: 200000000 + uint64(i), StreamValue: ToDecimal(decimal.NewFromInt(int64(200 + i)))}, //nolint:gosec // will never be > 4 + 3: &TimestampedStreamValue{ObservedAtNanoseconds: 300000000 + uint64(i), StreamValue: ToDecimal(decimal.NewFromInt(int64(300 + i)))}, //nolint:gosec // will never be > 4 + }} + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, outctx, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + require.Len(t, decoded.StreamAggregates, 3) + assert.Equal(t, &TimestampedStreamValue{ObservedAtNanoseconds: 100000002, StreamValue: ToDecimal(decimal.NewFromInt(int64(102)))}, decoded.StreamAggregates[1][llotypes.AggregatorMedian]) + assert.Equal(t, &TimestampedStreamValue{ObservedAtNanoseconds: 200000002, StreamValue: ToDecimal(decimal.NewFromInt(int64(202)))}, decoded.StreamAggregates[2][llotypes.AggregatorMedian]) + assert.Equal(t, &TimestampedStreamValue{ObservedAtNanoseconds: 300000002, StreamValue: ToDecimal(decimal.NewFromInt(int64(302)))}, decoded.StreamAggregates[3][llotypes.AggregatorMedian]) + }) + t.Run("copies forwards values from the last outcome if aggregation fails", func(t *testing.T) { + }) + t.Run("does not copy forwards values from the last outcome that are no longer in channel definitions", func(t *testing.T) { + }) + t.Run("copies forwards values from last outcome if old value was a different type", func(t *testing.T) { + }) + t.Run("copies forwards values from last outcome if old value had a newer timestamp", func(t *testing.T) { + }) + t.Run("replaces value with new aggregation output if timestamp is newer", func(t *testing.T) { + }) + }) + }) + t.Run("if previousOutcome is retired, returns outcome mas normal", func(t *testing.T) { + previousOutcome := Outcome{ + LifeCycleStage: llotypes.LifeCycleStage("retired"), + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: uint64(102030409 * time.Second), + 2: uint64(102030409 * time.Second), + }, + } + encodedPreviousOutcome, err := p.OutcomeCodec.Encode(previousOutcome) + require.NoError(t, err) + + aos := []types.AttributedObservation{} + for i := 0; i < 4; i++ { + obs := Observation{ + UnixTimestampNanoseconds: uint64(102030415 * time.Second), + } + encoded, err2 := p.ObservationCodec.Encode(obs) + require.NoError(t, err2) + aos = append(aos, + types.AttributedObservation{ + Observation: encoded, + Observer: commontypes.OracleID(i), //nolint:gosec // will never be > 4 + }) + } + outcome, err := p.Outcome(ctx, ocr3types.OutcomeContext{SeqNr: 2, PreviousOutcome: encodedPreviousOutcome}, types.Query{}, aos) + require.NoError(t, err) + + decoded, err := p.OutcomeCodec.Decode(outcome) + require.NoError(t, err) + + assert.Equal(t, uint64(102030415000000000), decoded.ObservationTimestampNanoseconds) + require.Len(t, decoded.ValidAfterNanoseconds, 2) + assert.Equal(t, uint64(102030409*time.Second), decoded.ValidAfterNanoseconds[1]) + assert.Equal(t, uint64(102030409*time.Second), decoded.ValidAfterNanoseconds[2]) + }) +} + +func Test_MakeChannelHash(t *testing.T) { + t.Run("hashes channel definitions", func(t *testing.T) { + defs := ChannelDefinitionWithID{ + ChannelID: 1, + ChannelDefinition: llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormat(1), + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}, {StreamID: 2, Aggregator: llotypes.AggregatorMedian}, {StreamID: 3, Aggregator: llotypes.AggregatorMedian}}, + Opts: []byte(`{}`), + }, + } + hash := MakeChannelHash(defs) + // NOTE: Breaking this test by changing the hash below may break existing running instances + assert.Equal(t, "c0b72f4acb79bb8f5075f979f86016a30159266a96870b1c617b44426337162a", fmt.Sprintf("%x", hash)) + }) + + t.Run("different channelID makes different hash", func(t *testing.T) { + def1 := ChannelDefinitionWithID{ChannelID: 1} + def2 := ChannelDefinitionWithID{ChannelID: 2} + + assert.NotEqual(t, MakeChannelHash(def1), MakeChannelHash(def2)) + }) + + t.Run("different report format makes different hash", func(t *testing.T) { + def1 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormatJSON, + }, + } + def2 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + ReportFormat: llotypes.ReportFormatEVMPremiumLegacy, + }, + } + + assert.NotEqual(t, MakeChannelHash(def1), MakeChannelHash(def2)) + }) + + t.Run("different streamIDs makes different hash", func(t *testing.T) { + def1 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}}, + }, + } + def2 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Streams: []llotypes.Stream{{StreamID: 2, Aggregator: llotypes.AggregatorMedian}}, + }, + } + + assert.NotEqual(t, MakeChannelHash(def1), MakeChannelHash(def2)) + }) + + t.Run("different aggregators makes different hash", func(t *testing.T) { + def1 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorMedian}}, + }, + } + def2 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Streams: []llotypes.Stream{{StreamID: 1, Aggregator: llotypes.AggregatorQuote}}, + }, + } + + assert.NotEqual(t, MakeChannelHash(def1), MakeChannelHash(def2)) + }) + + t.Run("different opts makes different hash", func(t *testing.T) { + def1 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Opts: []byte(`{"foo":"bar"}`), + }, + } + def2 := ChannelDefinitionWithID{ + ChannelDefinition: llotypes.ChannelDefinition{ + Opts: []byte(`{"foo":"baz"}`), + }, + } + + assert.NotEqual(t, MakeChannelHash(def1), MakeChannelHash(def2)) + }) +} + +func Test_Outcome_Methods(t *testing.T) { + t.Run("protocol version 0", func(t *testing.T) { + t.Run("IsReportable", func(t *testing.T) { + outcome := Outcome{} + cid := llotypes.ChannelID(1) + + // Not reportable if retired + outcome.LifeCycleStage = LifeCycleStageRetired + require.EqualError(t, outcome.IsReportable(cid, 0, 0), "ChannelID: 1; Reason: IsReportable=false; retired channel") + + // No channel definition with ID + outcome.LifeCycleStage = LifeCycleStageProduction + outcome.ObservationTimestampNanoseconds = uint64(time.Unix(1726670490, 0).UnixNano()) //nolint:gosec // time won't be negative + outcome.ChannelDefinitions = map[llotypes.ChannelID]llotypes.ChannelDefinition{} + require.EqualError(t, outcome.IsReportable(cid, 0, 0), "ChannelID: 1; Reason: IsReportable=false; no channel definition with this ID") + + // No ValidAfterNanoseconds yet + outcome.ChannelDefinitions = map[llotypes.ChannelID]llotypes.ChannelDefinition{ + cid: {}, + } + require.EqualError(t, outcome.IsReportable(cid, 0, 0), "ChannelID: 1; Reason: IsReportable=false; no ValidAfterNanoseconds entry yet, this must be a new channel") + + // ValidAfterNanoseconds is in the future + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: uint64(1726670491 * time.Second)} + require.EqualError(t, outcome.IsReportable(cid, 0, 0), "ChannelID: 1; Reason: ChannelID: 1; Reason: IsReportable=false; not valid yet (observationsTimestampSeconds=1726670490, validAfterSeconds=1726670491)") + + // ValidAfterSeconds=ObservationTimestampSeconds; IsReportable=false + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: uint64(1726670490 * time.Second)} + require.EqualError(t, outcome.IsReportable(cid, 0, 0), "ChannelID: 1; Reason: ChannelID: 1; Reason: IsReportable=false; not valid yet (observationsTimestampSeconds=1726670490, validAfterSeconds=1726670490)") + + // ValidAfterSeconds 0", func(t *testing.T) { + t.Run("IsReportable", func(t *testing.T) { + defaultMinReportInterval := uint64(100 * time.Millisecond) + + outcome := Outcome{} + cid := llotypes.ChannelID(1) + + // Not reportable if retired + outcome.LifeCycleStage = LifeCycleStageRetired + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; retired channel") + + obsTSNanos := uint64(time.Unix(1726670490, 1000).UnixNano()) //nolint:gosec // time won't be negative + + // No channel definition with ID + outcome.LifeCycleStage = LifeCycleStageProduction + outcome.ObservationTimestampNanoseconds = obsTSNanos + outcome.ChannelDefinitions = map[llotypes.ChannelID]llotypes.ChannelDefinition{} + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; no channel definition with this ID") + + // No ValidAfterNanoseconds yet + outcome.ChannelDefinitions[cid] = llotypes.ChannelDefinition{} + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; no ValidAfterNanoseconds entry yet, this must be a new channel") + + // ValidAfterNanoseconds is 1ns in the future; IsReportable=false + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos + 1} + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; not valid yet (ObservationTimestampNanoseconds=1726670490000001000, validAfterNanoseconds=1726670490000001001, minReportInterval=100000000); 0.100000 seconds (100000001ns) until reportable") + + // ValidAfterNanoseconds is 1s in the future; IsReportable=false + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos + uint64(1*time.Second)} + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; not valid yet (ObservationTimestampNanoseconds=1726670490000001000, validAfterNanoseconds=1726670491000001000, minReportInterval=100000000); 1.100000 seconds (1100000000ns) until reportable") + + // ValidAfterNanoseconds is 100ms-1ns in the past; IsReportable=false + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos - uint64(100*time.Millisecond) + 1} + require.EqualError(t, outcome.IsReportable(cid, 1, defaultMinReportInterval), "ChannelID: 1; Reason: IsReportable=false; not valid yet (ObservationTimestampNanoseconds=1726670490000001000, validAfterNanoseconds=1726670489900001001, minReportInterval=100000000); 0.000000 seconds (1ns) until reportable") + + // ValidAfterNanoseconds is exactly 100ms in the past; IsReportable=true + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos - uint64(100*time.Millisecond)} + require.Nil(t, outcome.IsReportable(cid, 1, defaultMinReportInterval)) + + // ValidAfterNanoseconds is 100ms+1ns in the past; IsReportable=true + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos - uint64(100*time.Millisecond) - 1} + require.Nil(t, outcome.IsReportable(cid, 1, defaultMinReportInterval)) + + // zero report cadence allows overlaps + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{cid: obsTSNanos} + require.Nil(t, outcome.IsReportable(cid, 1, 0)) + }) + t.Run("IsReportable with ReportFormatEVMPremiumLegacy", func(t *testing.T) { + outcome := Outcome{} + cid := llotypes.ChannelID(1) + + obsTSNanos := uint64(time.Unix(1726670490, 1e9-1).UnixNano()) //nolint:gosec // time won't be negative + + outcome.LifeCycleStage = LifeCycleStageProduction + outcome.ObservationTimestampNanoseconds = obsTSNanos + outcome.ChannelDefinitions = map[llotypes.ChannelID]llotypes.ChannelDefinition{ + cid: {ReportFormat: llotypes.ReportFormatEVMPremiumLegacy}, + } + outcome.ValidAfterNanoseconds = map[llotypes.ChannelID]uint64{ + cid: obsTSNanos - uint64(500*time.Millisecond), + } + + // if cadence is 0, but time is < 1s, does not report to avoid overlap + require.EqualError(t, outcome.IsReportable(cid, 1, uint64(0)), "ChannelID: 1; Reason: ChannelID: 1; Reason: IsReportable=false; not valid yet (observationsTimestampSeconds=1726670490, validAfterSeconds=1726670490)") + // if cadence is < 1s, if time is < 1s, does not report to avoid overlap + require.EqualError(t, outcome.IsReportable(cid, 1, uint64(100*time.Millisecond)), "ChannelID: 1; Reason: ChannelID: 1; Reason: IsReportable=false; not valid yet (observationsTimestampSeconds=1726670490, validAfterSeconds=1726670490)") + // if cadence is < 1s, if time is >= 1s, does report + outcome.ValidAfterNanoseconds[cid] = obsTSNanos - uint64(1*time.Second) + assert.Nil(t, outcome.IsReportable(cid, 1, uint64(100*time.Millisecond))) + // if cadence is exactly 1s, if time is >= 1s, does report + assert.Nil(t, outcome.IsReportable(cid, 1, uint64(1*time.Second))) + // if cadence is 5s, if time is < 5s, does not report because cadence hasn't elapsed + require.EqualError(t, outcome.IsReportable(cid, 1, uint64(5*time.Second)), "ChannelID: 1; Reason: IsReportable=false; not valid yet (ObservationTimestampNanoseconds=1726670490999999999, validAfterNanoseconds=1726670489999999999, minReportInterval=5000000000); 4.000000 seconds (4000000000ns) until reportable") + }) + t.Run("ReportableChannels", func(t *testing.T) { + defaultMinReportInterval := uint64(1 * time.Second) + + outcome := Outcome{ + ObservationTimestampNanoseconds: uint64(time.Unix(1726670490, 0).UnixNano()), //nolint:gosec // time won't be negative + ChannelDefinitions: map[llotypes.ChannelID]llotypes.ChannelDefinition{ + 1: {}, + 2: {}, + 3: {}, + }, + ValidAfterNanoseconds: map[llotypes.ChannelID]uint64{ + 1: uint64(1726670489 * time.Second), + 3: uint64(1726670489 * time.Second), + }, + } + reportable, unreportable := outcome.ReportableChannels(1, defaultMinReportInterval) + assert.Equal(t, []llotypes.ChannelID{1, 3}, reportable) + require.Len(t, unreportable, 1) + assert.Equal(t, "ChannelID: 2; Reason: IsReportable=false; no ValidAfterNanoseconds entry yet, this must be a new channel", unreportable[0].Error()) + }) + }) +} diff --git a/llo/protocol/calculated/calculated.go b/llo/protocol/calculated/calculated.go index 1f7d0046..c175930a 100644 --- a/llo/protocol/calculated/calculated.go +++ b/llo/protocol/calculated/calculated.go @@ -78,10 +78,9 @@ var defaultEnv = map[string]any{ "SMA": SMA, "WMA": WMA, "EMA": EMA, - // TWAP needs the round's observation timestamp to anchor its window, so - // NewEnv rebinds it per round. This default only reports that it was called - // against an environment NewEnv did not build. - "TWAP": twapUnbound, + // TWAP is a recognized DSL function that accepts a history window and a + // configuration map. The implementation is not provided in this package. + "TWAP": twapStub, // History is rewritten away at compile time (see history_ast.go). It is // registered only so that a call surviving to evaluation fails loudly // instead of resolving to an undefined identifier or, worse, to something @@ -95,6 +94,13 @@ func historyCallReached(...any) (decimal.Decimal, error) { return decimal.Decimal{}, fmt.Errorf("%s was not resolved at compile time; this is a bug in expression compilation", HistoryFunctionName) } +// twapStub is the runtime placeholder for TWAP. The function signature is +// kept so expressions referencing TWAP parse and compile; evaluation returns +// an error. +func twapStub(...any) (decimal.Decimal, error) { + return decimal.Decimal{}, errors.New("TWAP is not implemented") +} + var ( pool = sync.Pool{ New: func() any { @@ -188,10 +194,6 @@ func (e environment) release() { func NewEnv(observationTimestampNanoseconds uint64) environment { env := pool.Get().(environment) env["observations_timestamp"] = observationTimestampNanoseconds - // TWAP's window is anchored on the round's consensus observation timestamp, - // not on the data, so it is bound per round. release() restores the default - // binding, which fails if called. - env["TWAP"] = twapFunc(observationTimestampNanoseconds) return env } diff --git a/llo/protocol/calculated/decimalmath.go b/llo/protocol/calculated/decimalmath.go index 1b0c959a..4c348ff5 100644 --- a/llo/protocol/calculated/decimalmath.go +++ b/llo/protocol/calculated/decimalmath.go @@ -16,8 +16,6 @@ import ( // racing with a read can yield a corrupted value rather than merely a stale one — // which in a consensus path means two nodes disagreeing, or a panic. // -// TWAP makes it far more likely by calling ln and exp once per bucket. -// // The lock is taken per call rather than per evaluation to keep hold times short. // The cost is negligible against the arithmetic it guards. var transcendentalMu sync.Mutex @@ -173,8 +171,7 @@ func decimalToInt(name string, d decimal.Decimal, minimum, maximum int64) (int, // // 1. No float64. math.Log and math.Exp are not guaranteed bit-identical across // architectures or Go versions, so all logarithms and exponentials go through -// decimal.Ln and decimal.ExpTaylor at a fixed precision. This is why the TWAP -// implementation here is a port of the mercury float-based one, not a reuse. +// decimal.Ln and decimal.ExpTaylor at a fixed precision. // 2. No reliance on decimal.DivisionPrecision. That is a mutable package-level // global: anything in the process can change it and silently move every Div // result. Every division here passes an explicit precision (divRound). @@ -227,7 +224,7 @@ func ln(x decimal.Decimal) (decimal.Decimal, error) { // result, not of the input: exp(1e6) has ~434,000 digits and does not complete in // any useful time. Callers currently only pass logarithms of stored values, which // MaxDecimalExponent already bounds to about ±2302, so the limit is not reachable -// through TWAP today — it is here so that stays true if another caller appears. +// today — it is here so that stays true if a caller appears. func exp(x decimal.Decimal) (decimal.Decimal, error) { if x.Abs().GreaterThan(decimal.NewFromInt(maxExpArgument)) { return decimal.Decimal{}, fmt.Errorf("exponential argument %s exceeds the maximum magnitude of %d", x, maxExpArgument) diff --git a/llo/protocol/calculated/doc.go b/llo/protocol/calculated/doc.go index 13baf241..695cc447 100644 --- a/llo/protocol/calculated/doc.go +++ b/llo/protocol/calculated/doc.go @@ -25,8 +25,6 @@ // // Avg(History(s10001, 10)) // EMA(History(s10001, 50), 20) -// TWAP(History(s10001, 600), {window: Duration("5m"), minSamples: 240, -// maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30}) // // The call is both the declaration of how much history to persist and the read of // it. There is no separate configuration: the depth kept for a stream is the @@ -67,7 +65,7 @@ // SMA(w, n) simple mean of the newest n // WMA(w, n) linearly weighted, newest weighted n and the oldest of the n weighted 1 // EMA(w, n) seeded with the mean of the oldest n, then alpha = 2/(n+1) newest-ward -// TWAP(w, c) time-weighted average price over c.window, filling gaps by type +// TWAP(w, c) time-weighted average price (not implemented; recognized but not evaluated) // // A window may only be passed directly to one of these. Add(History(s1, 10), 2) is // rejected when the expression is validated, not left to fail during evaluation. diff --git a/llo/protocol/calculated/evaluate_fuzz_test.go b/llo/protocol/calculated/evaluate_fuzz_test.go index 60dcf6c3..42bc5286 100644 --- a/llo/protocol/calculated/evaluate_fuzz_test.go +++ b/llo/protocol/calculated/evaluate_fuzz_test.go @@ -55,7 +55,6 @@ func FuzzEvaluateExpression(f *testing.F) { "Sum(History(s1, 3))", "Min(History(s1, 3))", "Max(History(s1, 3))", - `TWAP(History(s1, 3), {window: Duration("3s"), minSamples: 1, maxHeadGap: 3, maxInteriorGap: 3, maxTailGap: 3})`, "Ln(s1)", "Log(s1, s2)", "Pow(s1, s2)", diff --git a/llo/protocol/calculated/functions_bench_test.go b/llo/protocol/calculated/functions_bench_test.go index acfa85b1..7cc5a02d 100644 --- a/llo/protocol/calculated/functions_bench_test.go +++ b/llo/protocol/calculated/functions_bench_test.go @@ -18,11 +18,6 @@ import ( // The figure to keep in mind is the round interval, on the order of a second: // every expression of every channel is evaluated inside one state transition, so // per-expression cost multiplies by the channel count. -// -// TWAP is the one to watch. It takes a logarithm per observed bucket and an -// exponential per bucket in the window, and all of those serialize on the -// process-wide transcendental lock (see decimalmath.go), so its cost does not -// parallelize across the plugin instances sharing a process. func benchSeries(depth int, intervalSeconds int) Series { values := make([]decimal.Decimal, 0, depth) @@ -73,134 +68,6 @@ func BenchmarkWindowFunctions(b *testing.B) { } } -// BenchmarkTWAP measures the settlement-window sizes an operator would actually -// configure. -func BenchmarkTWAP(b *testing.B) { - for _, windowSeconds := range []int{60, 300, 900} { - // One record per second, fully covering the window. - window := benchSeries(windowSeconds, 1) - anchorNs := uint64(windowSeconds+1) * uint64(time.Second) - cfg := map[string]any{ - "window": time.Duration(windowSeconds) * time.Second, - "minSamples": windowSeconds / 2, - "maxHeadGap": windowSeconds, - "maxInteriorGap": windowSeconds, - "maxTailGap": windowSeconds, - } - twap := twapFunc(anchorNs) - - b.Run(fmt.Sprintf("window=%ds", windowSeconds), func(b *testing.B) { - for range b.N { - if _, err := twap(window, cfg); err != nil { - b.Fatal(err) - } - } - }) - } -} - -// BenchmarkTWAPSparse is the worst case for the filling strategy: only interior -// interpolation needs log space, so cost scales with how much of the window is -// missing. A window observed every Nth second is the expensive shape. -func BenchmarkTWAPSparse(b *testing.B) { - const windowSeconds = 300 - - for _, everyNth := range []int{1, 2, 5, 30} { - depth := windowSeconds / everyNth - window := benchSeries(depth, everyNth) - anchorNs := uint64(windowSeconds+1) * uint64(time.Second) - cfg := map[string]any{ - "window": time.Duration(windowSeconds) * time.Second, - "minSamples": 1, - "maxHeadGap": windowSeconds, - "maxInteriorGap": windowSeconds, - "maxTailGap": windowSeconds, - } - twap := twapFunc(anchorNs) - - b.Run(fmt.Sprintf("observedEvery=%ds", everyNth), func(b *testing.B) { - for range b.N { - if _, err := twap(window, cfg); err != nil { - b.Fatal(err) - } - } - }) - } -} - -// BenchmarkTWAPRealisticGaps measures the worst case a production acceptance rule -// actually admits. -// -// The permissive thresholds in BenchmarkTWAPSparse exist to force interpolation -// and show where the cost lives; they are not deployable. With the spec's example -// thresholds (minSamples 240 of 300, maxInteriorGap 10) a window can be missing at -// most 60 buckets, so interpolation is bounded no matter how the gaps fall. -func BenchmarkTWAPRealisticGaps(b *testing.B) { - const windowSeconds = 300 - const minSamples = 240 - - // 240 observations in a 300-second window, with the 60 missing buckets spread - // as 20 interior gaps of 3 — within a maxInteriorGap of 10. - values := make([]decimal.Decimal, 0, minSamples) - timestamps := make([]uint64, 0, minSamples) - second := 0 - for len(values) < minSamples && second < windowSeconds { - if second%15 >= 12 { // 3 missing out of every 15 - second++ - continue - } - values = append(values, decimal.New(int64(110000000000000000+second), -8)) - timestamps = append(timestamps, uint64(second+1)*uint64(time.Second)) - second++ - } - window, err := NewSeries(values, timestamps) - if err != nil { - b.Fatal(err) - } - - cfg := map[string]any{ - "window": time.Duration(windowSeconds) * time.Second, - "minSamples": len(values), - "maxHeadGap": 10, - "maxInteriorGap": 10, - "maxTailGap": 10, - } - twap := twapFunc(uint64(windowSeconds+1) * uint64(time.Second)) - - b.ReportMetric(float64(windowSeconds-len(values)), "missingBuckets") - b.ResetTimer() - for range b.N { - if _, err := twap(window, cfg); err != nil { - b.Fatal(err) - } - } -} - -// BenchmarkTWAPParallel shows what the transcendental lock costs when several -// plugin instances evaluate TWAP at once. Compare ns/op against the serial -// benchmark: no speedup means the lock is the limit. -func BenchmarkTWAPParallel(b *testing.B) { - const windowSeconds = 300 - window := benchSeries(windowSeconds, 1) - anchorNs := uint64(windowSeconds+1) * uint64(time.Second) - cfg := map[string]any{ - "window": time.Duration(windowSeconds) * time.Second, - "minSamples": windowSeconds / 2, - "maxHeadGap": windowSeconds, - "maxInteriorGap": windowSeconds, - "maxTailGap": windowSeconds, - } - twap := twapFunc(anchorNs) - - b.RunParallel(func(pb *testing.PB) { - for pb.Next() { - if _, err := twap(window, cfg); err != nil { - b.Fatal(err) - } - } - }) -} - // BenchmarkProcessCalculatedStreams measures a whole round's expression work, // which is what the round budget actually has to absorb. func BenchmarkProcessCalculatedStreams(b *testing.B) { @@ -212,7 +79,6 @@ func BenchmarkProcessCalculatedStreams(b *testing.B) { {"scalar", "Add(s1, s2)", 0}, {"avg/depth=300", "Avg(History(s1, 300))", 300}, {"ema/depth=300", "EMA(History(s1, 300), 20)", 300}, - {"twap/window=300", `TWAP(History(s1, 300), {window: Duration("5m"), minSamples: 150, maxHeadGap: 300, maxInteriorGap: 300, maxTailGap: 300})`, 300}, } { for _, channels := range []int{1, 32} { b.Run(fmt.Sprintf("%s/channels=%d", tc.name, channels), func(b *testing.B) { diff --git a/llo/protocol/calculated/functions_twap.go b/llo/protocol/calculated/functions_twap.go deleted file mode 100644 index c5686f05..00000000 --- a/llo/protocol/calculated/functions_twap.go +++ /dev/null @@ -1,455 +0,0 @@ -package calculated - -import ( - "errors" - "fmt" - "sort" - "strings" - "time" - - "github.com/shopspring/decimal" -) - -// TWAP is ported from the original spec (ADR 0013/0014/0015), semantics are unchanged. -// It is a port rather than a reuse for two reasons: the source works from decoded -// values and a clock, while here the input is an already-agreed history window; -// and the source computes in float64, which is not guaranteed bit-identical -// across architectures and so cannot appear in a consensus path. Every logarithm, -// exponential and division below is decimal at a fixed precision. -var ( - // ErrTWAPRejected is what every rejection satisfies errors.Is against, so - // callers can detect a rejected window without inspecting the reasons. - ErrTWAPRejected = errors.New("TWAP window rejected by the acceptance rule") - - // ErrTWAPConfig is returned for a malformed configuration. Configuration is - // static, so this is a deployment error rather than a data condition. - ErrTWAPConfig = errors.New("invalid TWAP configuration") -) - -// TWAPRejectionReason enumerates why a window failed the acceptance rule. A -// window can fail several checks at once. -type TWAPRejectionReason string - -const ( - // ReasonInsufficientSamples: M < minSamples, the coverage floor. - ReasonInsufficientSamples TWAPRejectionReason = "min_samples" - // ReasonHeadGapTooLong: Ghead > maxHeadGap, backfilled prefix too long. - ReasonHeadGapTooLong TWAPRejectionReason = "head_gap_too_long" - // ReasonInteriorGapTooLong: Gint > maxInteriorGap, longest both-sides-anchored gap too long. - ReasonInteriorGapTooLong TWAPRejectionReason = "interior_gap_too_long" - // ReasonTailGapTooLong: Gtail > maxTailGap, carry-forward suffix too long. - ReasonTailGapTooLong TWAPRejectionReason = "tail_gap_too_long" -) - -// TWAPRejection carries the measured statistics alongside the thresholds they -// failed, so an operator can tell a thin window from a stalled feed without -// reproducing the calculation. -type TWAPRejection struct { - Reasons []TWAPRejectionReason - M, Ghead, Gint, Gtail int - MinSamples, MaxHeadGap, MaxInteriorGap, MaxTailGap int - WindowStartSeconds, WindowEndSeconds int64 - Records int -} - -func (e *TWAPRejection) Error() string { - reasons := make([]string, 0, len(e.Reasons)) - for _, reason := range e.Reasons { - reasons = append(reasons, string(reason)) - } - return fmt.Sprintf("TWAP: window [%d, %d) rejected (%s): M=%d/%d Ghead=%d/%d Gint=%d/%d Gtail=%d/%d from %d records", - e.WindowStartSeconds, e.WindowEndSeconds, strings.Join(reasons, ","), - e.M, e.MinSamples, e.Ghead, e.MaxHeadGap, e.Gint, e.MaxInteriorGap, e.Gtail, e.MaxTailGap, e.Records) -} - -func (e *TWAPRejection) Is(target error) bool { return target == ErrTWAPRejected } - -// twapConfig is the acceptance rule for one window size. Every field is required: -// a defaulted threshold would silently accept a window an operator never approved. -type twapConfig struct { - windowSeconds int64 - minSamples int - maxHeadGap int - maxInteriorGap int - maxTailGap int -} - -// twapBucket is one second of the dense series the specification operates on. -// price is only meaningful when observed is true. -// -// The specification is written in log-price space throughout: build X[i] = ln(P[i]), -// fill gaps in that space, then average exp of the filled series. This stores the -// price instead, and moves into log space only where filling actually requires it. -// -// That is not a shortcut, it is the same series. For an observed bucket the -// specification computes exp(ln(P)) = P. For a head gap it backfills the first -// observed log-price, and for a tail gap it carries the last, so exponentiating -// those yields that same observed price. Only interior interpolation produces a -// value that is not already a price. -// -// The reason it matters is cost. Filling in log space needs a logarithm per -// observed bucket and an exponential per bucket in the window — about 600 -// operations for a five-minute window — and they all serialize on the -// transcendental lock. Measured: 197ms per evaluation for a 300-second window, -// and 7.2s for 32 such channels in one round, against a round budget on the -// order of a second. Doing it this way, a fully covered window needs no -// transcendental operations at all, and a window with gaps needs two logarithms -// per gap plus one exponential per missing bucket. -type twapBucket struct { - observed bool - price decimal.Decimal -} - -// twapFunc returns the TWAP function bound to a round's consensus observation -// timestamp, which anchors the window. -// -// The anchor has to come from the round rather than from the data: taking it from -// the newest record would silently shorten the window whenever a feed stalled, -// which is exactly the condition the acceptance rule exists to catch. -func twapFunc(observationTimestampNanoseconds uint64) func(any, any) (decimal.Decimal, error) { - return func(x any, rawConfig any) (decimal.Decimal, error) { - series, err := window("TWAP", x) - if err != nil { - return decimal.Decimal{}, err - } - cfg, err := parseTWAPConfig(rawConfig) - if err != nil { - return decimal.Decimal{}, err - } - // NOTE: whether the requested history depth can ever supply minSamples - // observations is a static property of the configuration, and is checked - // at configuration time rather than here. Checking it here would turn a - // specification-defined rejection (M < minSamples, a data condition with - // diagnostics) into a configuration error, losing the measured - // statistics an operator needs. - return twap(series, cfg, int64(observationTimestampNanoseconds/uint64(time.Second))) - } -} - -// twapUnbound is the default TWAP binding. NewEnv replaces it with a function -// bound to the round's observation timestamp; reaching this one means TWAP was -// called against an environment that was not built by NewEnv. -func twapUnbound(any, any) (decimal.Decimal, error) { - return decimal.Decimal{}, errors.New("TWAP has no observation timestamp bound; the environment was not created by NewEnv") -} - -func twap(series Series, cfg twapConfig, anchorSeconds int64) (decimal.Decimal, error) { - windowStart := anchorSeconds - cfg.windowSeconds - buckets := make([]twapBucket, cfg.windowSeconds) - - values, timestamps := series.Values(), series.Timestamps() - for i, ts := range timestamps { - seconds := int64(ts / uint64(time.Second)) - if seconds < windowStart || seconds >= anchorSeconds { - continue // outside the half-open window (ADR 0013) - } - // The price must be positive: the filling rules are defined in log space, - // so a non-positive price has no representation there. Checked here for - // every observed bucket rather than only where a logarithm is taken, so - // acceptance does not depend on where the gaps happen to fall. - if !values[i].IsPositive() { - return decimal.Decimal{}, fmt.Errorf("TWAP: record %d: price %s must be positive", i, values[i]) - } - // Timestamps are strictly increasing, so a later record legitimately - // overwrites an earlier one in the same bucket: newest wins. - buckets[seconds-windowStart] = twapBucket{observed: true, price: values[i]} - } - - m, gHead, gInt, gTail := twapGapStats(buckets) - - var reasons []TWAPRejectionReason - // A floor of 1 observation is required for head backfill to have an anchor. - // With a validated minSamples >= 1 this is redundant, but it keeps a - // misconfiguration from reaching an out-of-range index below. - minSamples := max(cfg.minSamples, 1) - if m < minSamples { - reasons = append(reasons, ReasonInsufficientSamples) - } - if gHead > cfg.maxHeadGap { - reasons = append(reasons, ReasonHeadGapTooLong) - } - if gInt > cfg.maxInteriorGap { - reasons = append(reasons, ReasonInteriorGapTooLong) - } - if gTail > cfg.maxTailGap { - reasons = append(reasons, ReasonTailGapTooLong) - } - if len(reasons) > 0 { - return decimal.Decimal{}, &TWAPRejection{ - Reasons: reasons, - M: m, Ghead: gHead, Gint: gInt, Gtail: gTail, - MinSamples: cfg.minSamples, MaxHeadGap: cfg.maxHeadGap, - MaxInteriorGap: cfg.maxInteriorGap, MaxTailGap: cfg.maxTailGap, - WindowStartSeconds: windowStart, WindowEndSeconds: anchorSeconds, - Records: series.Len(), - } - } - - return twapFillThenAverage(buckets) -} - -// twapGapStats measures M, Ghead, Gint and Gtail by classifying each missing run -// by its position (spec §2, ADR 0015). -// -// Ghead and Gtail are kept separate from Gint deliberately: Gint is the -// both-sides-anchored statistic, and a head or tail run has only one anchor. A -// run spanning the whole window is classified as none of them because it has no -// anchors at all; such a window is always rejected by the M check. -func twapGapStats(buckets []twapBucket) (m, gHead, gInt, gTail int) { - n := len(buckets) - for i := 0; i < n; { - runStart := i - observed := buckets[i].observed - for i < n && buckets[i].observed == observed { - i++ - } - runLen := i - runStart - - if observed { - m += runLen - continue - } - switch { - case runStart == 0 && i == n: - // Entire window missing: no anchors, so not head, tail or interior. - case runStart == 0: - gHead = runLen - case i == n: - gTail = runLen - default: - gInt = max(gInt, runLen) - } - } - return m, gHead, gInt, gTail -} - -// twapFillThenAverage fills every bucket per spec §4 and returns the mean price -// over the full window. -// -// Callers must only reach this once the acceptance rule has passed, which -// guarantees at least one observation. -func twapFillThenAverage(buckets []twapBucket) (decimal.Decimal, error) { - n := len(buckets) - filled := make([]decimal.Decimal, n) - - for i := 0; i < n; { - if buckets[i].observed { - filled[i] = buckets[i].price // spec §4.1: X[i] passes through - i++ - continue - } - runStart := i - for i < n && !buckets[i].observed { - i++ - } - switch { - case runStart == 0: - // Head gap: backfill the first observed price (ADR 0015). - // buckets[i] is observed, because a window with no observation at - // all was rejected above. - for k := 0; k < i; k++ { - filled[k] = buckets[i].price - } - case i == n: - // Tail gap: carry forward the last observed price (spec §4.3). - for k := runStart; k < n; k++ { - filled[k] = buckets[runStart-1].price - } - default: - // Interior gap: log-linear interpolation between the bracketing - // anchors at runStart-1 and i (spec §4.2). This is the only case - // that needs log space, so it is the only one that pays for it. - if err := twapInterpolate(buckets, filled, runStart, i); err != nil { - return decimal.Decimal{}, err - } - } - } - - // TWAP = mean over N (spec §4-5, denominator N not M). - sum := decimal.Zero - for _, price := range filled { - sum = sum.Add(price) - } - return divRoundByInt(sum, n, precision) -} - -// twapInterpolate fills the missing run [runStart, rightIdx) between its -// bracketing anchors (spec §4.2). -// -// Linear interpolation in log space is geometric interpolation in price space: a -// gap between 100 and 1600 fills as 200, 400, 800, not as evenly spaced prices. -// So rather than exponentiating each interpolated log-price, this takes the -// constant per-second ratio once and steps through the gap by multiplication: -// -// ratio = (right / left) ^ (1 / span) -// filled[k] = filled[k-1] * ratio -// -// One power per gap instead of two logarithms plus one exponential per missing -// bucket. With the spec's example thresholds a window can be missing 60 buckets, -// which cost ~73ms the other way and a fraction of that here. Exponentials are the -// expensive operation (~0.5ms each) and reducing their precision only helps by -// about a factor of two, so cutting their number is the only lever that matters. -// -// Determinism: the ratio is computed at a fixed precision and every step is -// rounded, so the sequence is reproducible — the same requirement EMA has, for the -// same reason. -func twapInterpolate(buckets []twapBucket, filled []decimal.Decimal, runStart, rightIdx int) error { - leftIdx := runStart - 1 - left, right := buckets[leftIdx].price, buckets[rightIdx].price - - growth, err := divRound(right, left, doublePrecision) - if err != nil { - return fmt.Errorf("TWAP: bucket %d: %w", leftIdx, err) - } - exponent, err := divRoundByInt(decimal.NewFromInt(1), rightIdx-leftIdx, doublePrecision) - if err != nil { - return err - } - ratio, err := decimalPow(growth, exponent, doublePrecision) - if err != nil { - return fmt.Errorf("TWAP: interpolating buckets %d..%d: %w", runStart, rightIdx-1, err) - } - - price := left - for k := runStart; k < rightIdx; k++ { - price = price.Mul(ratio).Round(doublePrecision) - filled[k] = price - } - return nil -} - -// parseTWAPConfig decodes and validates the configuration map. -// -// Every key is required and no key is optional: a defaulted threshold would mean -// accepting a window against a rule nobody wrote down. Unknown keys are rejected -// too, so a typo fails loudly instead of leaving a threshold at its intended -// value by accident. -func parseTWAPConfig(raw any) (twapConfig, error) { - fields, ok := raw.(map[string]any) - if !ok { - return twapConfig{}, fmt.Errorf("%w: expected a configuration map, got %T", ErrTWAPConfig, raw) - } - - const ( - keyWindow = "window" - keyMinSamples = "minSamples" - keyMaxHeadGap = "maxHeadGap" - keyMaxInteriorGap = "maxInteriorGap" - keyMaxTailGap = "maxTailGap" - ) - known := map[string]bool{ - keyWindow: true, keyMinSamples: true, keyMaxHeadGap: true, - keyMaxInteriorGap: true, keyMaxTailGap: true, - } - unknown := make([]string, 0) - for key := range fields { - if !known[key] { - unknown = append(unknown, key) - } - } - if len(unknown) > 0 { - sort.Strings(unknown) - return twapConfig{}, fmt.Errorf("%w: unknown keys %s", ErrTWAPConfig, strings.Join(unknown, ", ")) - } - - windowSeconds, err := twapWindowSeconds(fields[keyWindow]) - if err != nil { - return twapConfig{}, err - } - minSamples, err := twapConfigInt(fields, keyMinSamples, 1) - if err != nil { - return twapConfig{}, err - } - maxHeadGap, err := twapConfigInt(fields, keyMaxHeadGap, 0) - if err != nil { - return twapConfig{}, err - } - maxInteriorGap, err := twapConfigInt(fields, keyMaxInteriorGap, 0) - if err != nil { - return twapConfig{}, err - } - maxTailGap, err := twapConfigInt(fields, keyMaxTailGap, 0) - if err != nil { - return twapConfig{}, err - } - - if int64(minSamples) > windowSeconds { - return twapConfig{}, fmt.Errorf("%w: minSamples %d exceeds the %d one-second buckets in the window", - ErrTWAPConfig, minSamples, windowSeconds) - } - - return twapConfig{ - windowSeconds: windowSeconds, - minSamples: minSamples, - maxHeadGap: maxHeadGap, - maxInteriorGap: maxInteriorGap, - maxTailGap: maxTailGap, - }, nil -} - -// twapWindowSeconds resolves the window length, which must be a whole number of -// seconds because the calculation is defined over one-second buckets. -func twapWindowSeconds(raw any) (int64, error) { - if raw == nil { - return 0, fmt.Errorf("%w: window is required", ErrTWAPConfig) - } - - var nanoseconds decimal.Decimal - switch v := raw.(type) { - case time.Duration: - nanoseconds = decimal.NewFromInt(int64(v)) - default: - d, err := toDecimal(raw) - if err != nil { - return 0, fmt.Errorf("%w: window: %s", ErrTWAPConfig, err) - } - nanoseconds = d - } - - perSecond := decimal.NewFromInt(int64(time.Second)) - if !nanoseconds.Mod(perSecond).IsZero() { - return 0, fmt.Errorf("%w: window must be a whole number of seconds", ErrTWAPConfig) - } - // DivRound rather than Div: Div reads the mutable decimal.DivisionPrecision - // global. The division is exact here, but the rule holds everywhere. - // Compared and bounded as a decimal, before any narrowing; see decimalToInt. - // The upper bound also caps the per-evaluation work: the calculation - // allocates and fills one bucket per second of the window. - secondsDecimal := nanoseconds.DivRound(perSecond, 0) - if secondsDecimal.LessThan(decimal.NewFromInt(1)) { - return 0, fmt.Errorf("%w: window must be at least one second, got %s", ErrTWAPConfig, secondsDecimal) - } - if secondsDecimal.GreaterThan(decimal.NewFromInt(twapMaxWindowSeconds)) { - return 0, fmt.Errorf("%w: window of %s seconds exceeds the maximum of %d", - ErrTWAPConfig, secondsDecimal, twapMaxWindowSeconds) - } - seconds, err := decimalToInt("window", secondsDecimal, 1, twapMaxWindowSeconds) - if err != nil { - return 0, fmt.Errorf("%w: %s", ErrTWAPConfig, err) - } - return int64(seconds), nil -} - -// twapMaxWindowSeconds bounds the number of one-second buckets a single TWAP -// evaluation may allocate and fill. 24 hours is far beyond any settlement window -// while keeping the per-round work bounded. -const twapMaxWindowSeconds = 24 * 60 * 60 - -func twapConfigInt(fields map[string]any, key string, minimum int) (int, error) { - raw, ok := fields[key] - if !ok || raw == nil { - return 0, fmt.Errorf("%w: %s is required", ErrTWAPConfig, key) - } - d, err := toDecimal(raw) - if err != nil { - return 0, fmt.Errorf("%w: %s: %s", ErrTWAPConfig, key, err) - } - // Bounded as a decimal before narrowing; see decimalToInt. The upper bound - // is the window cap, since every one of these counts seconds or samples - // inside a window that cannot itself be longer than that. - value, err := decimalToInt(key, d, int64(minimum), twapMaxWindowSeconds) - if err != nil { - return 0, fmt.Errorf("%w: %s", ErrTWAPConfig, err) - } - return value, nil -} diff --git a/llo/protocol/calculated/functions_twap_test.go b/llo/protocol/calculated/functions_twap_test.go deleted file mode 100644 index 7b68817f..00000000 --- a/llo/protocol/calculated/functions_twap_test.go +++ /dev/null @@ -1,468 +0,0 @@ -package calculated - -import ( - "errors" - "fmt" - "testing" - "time" - - "github.com/shopspring/decimal" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -// twapWindowStartSeconds is the arbitrary window start the ported cases use. -const twapWindowStartSeconds = 100 - -// twapSeries builds a window from one price per bucket, 0 meaning "no observation -// in that second". Buckets are one second apart starting at twapWindowStartSeconds. -// -// This mirrors the mercury calculator tests, where a report marks exactly one -// bucket observed. -func twapSeries(t *testing.T, prices []int64) Series { - t.Helper() - values := make([]decimal.Decimal, 0, len(prices)) - timestamps := make([]uint64, 0, len(prices)) - for i, price := range prices { - if price == 0 { - continue // missing observation - } - values = append(values, decimal.NewFromInt(price)) - timestamps = append(timestamps, uint64(twapWindowStartSeconds+i)*uint64(time.Second)) - } - s, err := NewSeries(values, timestamps) - require.NoError(t, err) - return s -} - -// twapConfigMap is the configuration as an expression would supply it. -func twapConfigMap(windowSeconds, minSamples, maxHeadGap, maxInteriorGap, maxTailGap int) map[string]any { - return map[string]any{ - "window": time.Duration(windowSeconds) * time.Second, - "minSamples": minSamples, - "maxHeadGap": maxHeadGap, - "maxInteriorGap": maxInteriorGap, - "maxTailGap": maxTailGap, - } -} - -// assertClose compares against a hand-computed value with a tolerance. -// -// Exactness is not available here and that is inherent to the algorithm, not a -// shortcut: the specification fills gaps in log-price space, so every bucket goes -// through exp(ln(price)). At any finite precision that round trip is not the -// identity, so a whole-number expectation cannot be matched bit-for-bit. The -// tolerance is many orders of magnitude tighter than any reporting precision. -func assertClose(t *testing.T, want string, got decimal.Decimal) { - t.Helper() - expected, err := decimal.NewFromString(want) - require.NoError(t, err) - - const tolerance = "0.000000000001" // 1e-12 - limit, err := decimal.NewFromString(tolerance) - require.NoError(t, err) - - diff := got.Sub(expected).Abs() - assert.True(t, diff.LessThanOrEqual(limit), - "expected %s (±%s), got %s (off by %s)", want, tolerance, got, diff) -} - -// TestTWAP_FillThenAverage is the mercury TestCalculate_FillThenAverage suite, -// ported case for case. The expected values are the same, which is the point: the -// port changed the arithmetic from float64 to decimal, not the semantics. -func TestTWAP_FillThenAverage(t *testing.T) { - t.Parallel() - - for _, tc := range []struct { - name string - prices []int64 // one entry per bucket; 0 = missing - minSamples int - maxHead int - maxInterior int - maxTail int - want string - wantReasons []TWAPRejectionReason - }{ - { - name: "no gaps: plain average over the full window", - prices: []int64{100, 200, 300}, - minSamples: 3, - want: "200", - }, - { - name: "interior gap at the threshold: log-linear interpolated", - prices: []int64{100, 0, 0, 0, 1600}, // Gint=3 - minSamples: 2, maxInterior: 3, - // Interpolating in log space doubles each step: 100,200,400,800,1600. - want: "620", - }, - { - name: "interior gap one over the threshold: rejected", - prices: []int64{100, 0, 0, 0, 1600}, - minSamples: 2, maxInterior: 2, - wantReasons: []TWAPRejectionReason{ReasonInteriorGapTooLong}, - }, - { - name: "tail gap at the threshold: carried forward from the last observed price", - prices: []int64{100, 200, 300, 0, 0}, // Gtail=2 - minSamples: 3, maxTail: 2, - want: "240", - }, - { - name: "tail gap one over the threshold: rejected", - prices: []int64{100, 200, 300, 0, 0}, - minSamples: 3, maxTail: 1, - wantReasons: []TWAPRejectionReason{ReasonTailGapTooLong}, - }, - { - name: "insufficient samples: rejected even though every gap is within its threshold", - prices: []int64{100, 0, 300, 0, 500}, // M=3, Gint=1 - minSamples: 4, maxInterior: 5, maxTail: 5, - wantReasons: []TWAPRejectionReason{ReasonInsufficientSamples}, - }, - { - name: "head gap at the threshold: backfilled from the first observed price", - prices: []int64{0, 0, 200, 300, 400}, // Ghead=2 - minSamples: 3, maxHead: 2, - want: "260", - }, - { - name: "head gap one over the threshold: rejected", - prices: []int64{0, 0, 200, 300, 400}, - minSamples: 3, maxHead: 1, - wantReasons: []TWAPRejectionReason{ReasonHeadGapTooLong}, - }, - { - // Gint is the both-sides-anchored statistic, so a head run must not - // count toward it. Ghead=2 while maxInterior is 1: if the head run - // leaked into Gint this would reject instead of producing a value. - name: "head run is not counted toward the interior-gap threshold", - prices: []int64{0, 0, 100, 0, 400}, // Ghead=2, Gint=1 - minSamples: 2, maxHead: 2, maxInterior: 1, - want: "180", - }, - { - name: "head and tail gap in the same window, both at their thresholds", - prices: []int64{0, 0, 100, 100, 0, 0}, // Ghead=2, Gtail=2 - minSamples: 2, maxHead: 2, maxTail: 2, - want: "100", - }, - { - name: "multiple applicable reasons are all returned, not just the first", - prices: []int64{100, 0, 0, 0, 0}, // M=1, Gtail=4 - minSamples: 3, maxInterior: 5, maxTail: 2, - wantReasons: []TWAPRejectionReason{ReasonInsufficientSamples, ReasonTailGapTooLong}, - }, - { - name: "head and tail reasons are reported alongside insufficient samples", - prices: []int64{0, 0, 0, 100, 0}, // M=1, Ghead=3, Gtail=1 - minSamples: 3, maxHead: 2, - wantReasons: []TWAPRejectionReason{ReasonInsufficientSamples, ReasonHeadGapTooLong, ReasonTailGapTooLong}, - }, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - - windowSeconds := len(tc.prices) - series := twapSeries(t, tc.prices) - cfg := twapConfigMap(windowSeconds, tc.minSamples, tc.maxHead, tc.maxInterior, tc.maxTail) - - // The anchor is the exclusive end of the window. - anchorNs := uint64(twapWindowStartSeconds+windowSeconds) * uint64(time.Second) - got, err := twapFunc(anchorNs)(series, cfg) - - if tc.wantReasons != nil { - require.Error(t, err) - require.ErrorIs(t, err, ErrTWAPRejected) - var rejection *TWAPRejection - require.ErrorAs(t, err, &rejection) - assert.Equal(t, tc.wantReasons, rejection.Reasons) - return - } - require.NoError(t, err) - assertClose(t, tc.want, got) - }) - } -} - -// TestTWAP_GapStats ports the mercury gap-classification cases. These are the -// statistics the acceptance rule is written in terms of, so they are worth -// pinning independently of the averaging. -func TestTWAP_GapStats(t *testing.T) { - t.Parallel() - - // o marks an observed bucket, x a missing one. - const o, x = true, false - - for _, tc := range []struct { - name string - observed []bool - wantM, wantHead, wantInt, wantTail int - }{ - {"all observed", []bool{o, o, o}, 3, 0, 0, 0}, - {"head gap only", []bool{x, x, o, o, o}, 3, 2, 0, 0}, - {"tail gap only", []bool{o, o, o, x, x}, 3, 0, 0, 2}, - {"single interior gap", []bool{o, x, o}, 2, 0, 1, 0}, - {"head and tail gaps", []bool{x, x, o, x, x}, 1, 2, 0, 2}, - {"head and interior gaps", []bool{x, o, x, x, o}, 2, 1, 2, 0}, - {"interior and tail gaps", []bool{o, x, x, x, x}, 1, 0, 0, 4}, - // No anchors at all, so no run is classified; the M check rejects it. - {"no observations", []bool{x, x, x, x, x}, 0, 0, 0, 0}, - {"single observed bucket", []bool{o}, 1, 0, 0, 0}, - {"single missing bucket", []bool{x}, 0, 0, 0, 0}, - {"alternating", []bool{o, x, o, x, o}, 3, 0, 1, 0}, - {"two interior gaps takes the longest", []bool{o, x, o, x, x, o}, 3, 0, 2, 0}, - {"three interior gaps increasing", []bool{o, x, o, x, x, o, x, x, x, o}, 4, 0, 3, 0}, - {"gaps not in order", []bool{o, x, x, x, o, x, o, x, x, o}, 4, 0, 3, 0}, - {"long head gap", []bool{x, x, x, x, x, o, o}, 2, 5, 0, 0}, - {"long tail gap", []bool{o, o, x, x, x, x, x}, 2, 0, 0, 5}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Parallel() - buckets := make([]twapBucket, len(tc.observed)) - for i, observed := range tc.observed { - buckets[i] = twapBucket{observed: observed, price: decimal.NewFromInt(1)} - } - m, head, interior, tail := twapGapStats(buckets) - assert.Equal(t, tc.wantM, m, "M") - assert.Equal(t, tc.wantHead, head, "Ghead") - assert.Equal(t, tc.wantInt, interior, "Gint") - assert.Equal(t, tc.wantTail, tail, "Gtail") - }) - } -} - -// TestTWAP_HalfOpenWindow covers ADR 0013: the anchor second belongs to the next -// window, and observations before the window start are dropped. -func TestTWAP_HalfOpenWindow(t *testing.T) { - t.Parallel() - - const windowSeconds = 3 - anchorSeconds := uint64(twapWindowStartSeconds + windowSeconds) - cfg := twapConfigMap(windowSeconds, 1, windowSeconds, windowSeconds, windowSeconds) - - // An observation exactly at the anchor is excluded, leaving the window empty. - series, err := NewSeries( - []decimal.Decimal{decimal.NewFromInt(100)}, - []uint64{anchorSeconds * uint64(time.Second)}, - ) - require.NoError(t, err) - _, err = twapFunc(anchorSeconds*uint64(time.Second))(series, cfg) - require.ErrorIs(t, err, ErrTWAPRejected, "the anchor second belongs to the next window") - - // One second earlier is inside the window. - series, err = NewSeries( - []decimal.Decimal{decimal.NewFromInt(100)}, - []uint64{(anchorSeconds - 1) * uint64(time.Second)}, - ) - require.NoError(t, err) - got, err := twapFunc(anchorSeconds*uint64(time.Second))(series, cfg) - require.NoError(t, err) - assertClose(t, "100", got) - - // An observation before the window start is dropped, so only the in-window - // one counts. - series, err = NewSeries( - []decimal.Decimal{decimal.NewFromInt(999), decimal.NewFromInt(100)}, - []uint64{(twapWindowStartSeconds - 5) * uint64(time.Second), (anchorSeconds - 1) * uint64(time.Second)}, - ) - require.NoError(t, err) - got, err = twapFunc(anchorSeconds*uint64(time.Second))(series, cfg) - require.NoError(t, err) - assertClose(t, "100", got) -} - -// TestTWAP_Deterministic is the property that makes TWAP usable in a consensus -// path: identical inputs give an identical result, with no float64 anywhere. -func TestTWAP_Deterministic(t *testing.T) { - t.Parallel() - - prices := []int64{1234, 0, 1240, 1250, 0, 0, 1300, 1310} - series := twapSeries(t, prices) - cfg := twapConfigMap(len(prices), 3, 2, 2, 2) - anchorNs := uint64(twapWindowStartSeconds+len(prices)) * uint64(time.Second) - - first, err := twapFunc(anchorNs)(series, cfg) - require.NoError(t, err) - for range 30 { - again, err := twapFunc(anchorNs)(series, cfg) - require.NoError(t, err) - require.True(t, first.Equal(again), "TWAP drifted: %s vs %s", first, again) - } -} - -func TestTWAP_ConfigValidation(t *testing.T) { - t.Parallel() - - series := twapSeries(t, []int64{100, 200, 300}) - anchorNs := uint64(twapWindowStartSeconds+3) * uint64(time.Second) - call := func(cfg any) error { - _, err := twapFunc(anchorNs)(series, cfg) - return err - } - - t.Run("every key is required", func(t *testing.T) { - t.Parallel() - // A defaulted threshold would mean accepting a window against a rule - // nobody wrote down. - for _, missing := range []string{"window", "minSamples", "maxHeadGap", "maxInteriorGap", "maxTailGap"} { - cfg := twapConfigMap(3, 1, 0, 0, 0) - delete(cfg, missing) - err := call(cfg) - require.ErrorIs(t, err, ErrTWAPConfig, "missing %s", missing) - require.ErrorContains(t, err, missing) - } - }) - - t.Run("unknown keys are rejected", func(t *testing.T) { - t.Parallel() - cfg := twapConfigMap(3, 1, 0, 0, 0) - cfg["maxHeadGapp"] = 1 - err := call(cfg) - require.ErrorIs(t, err, ErrTWAPConfig) - require.ErrorContains(t, err, "maxHeadGapp") - }) - - t.Run("window must be whole seconds and positive", func(t *testing.T) { - t.Parallel() - cfg := twapConfigMap(3, 1, 0, 0, 0) - cfg["window"] = 1500 * time.Millisecond - require.ErrorContains(t, call(cfg), "whole number of seconds") - - cfg["window"] = time.Duration(0) - require.ErrorContains(t, call(cfg), "at least one second") - - cfg["window"] = 48 * time.Hour - require.ErrorContains(t, call(cfg), "exceeds the maximum") - }) - - t.Run("oversized configuration values are rejected, not wrapped", func(t *testing.T) { - t.Parallel() - // decimal.IntPart narrows through int64 and returns the low 64 bits of - // an oversized value, so 2^64+1 comes back as 1. Bounding after that - // would silently accept a one-second window or a minSamples of 1. TWAP - // configuration is not required to be literal (see checkTWAP), so these - // values can come from stream data, bounded only by MaxDecimalExponent. - wrapped := decimal.RequireFromString("18446744073709551617") - - cfg := twapConfigMap(3, 1, 0, 0, 0) - // A whole number of seconds, so it clears the modulo check first. - cfg["window"] = wrapped.Mul(decimal.NewFromInt(int64(time.Second))) - require.ErrorContains(t, call(cfg), "exceeds the maximum") - - for _, key := range []string{"minSamples", "maxHeadGap", "maxInteriorGap", "maxTailGap"} { - cfg := twapConfigMap(3, 1, 0, 0, 0) - cfg[key] = wrapped - err := call(cfg) - require.ErrorIs(t, err, ErrTWAPConfig, key) - require.ErrorContains(t, err, "at most", key) - } - }) - - t.Run("thresholds must be whole and non-negative", func(t *testing.T) { - t.Parallel() - cfg := twapConfigMap(3, 1, 0, 0, 0) - cfg["maxHeadGap"] = -1 - require.ErrorContains(t, call(cfg), "at least 0") - - cfg = twapConfigMap(3, 1, 0, 0, 0) - cfg["minSamples"] = 0 - require.ErrorContains(t, call(cfg), "at least 1") - - cfg = twapConfigMap(3, 1, 0, 0, 0) - cfg["maxTailGap"] = 1.5 - require.ErrorContains(t, call(cfg), "whole number") - }) - - t.Run("minSamples cannot exceed the window", func(t *testing.T) { - t.Parallel() - require.ErrorContains(t, call(twapConfigMap(3, 4, 0, 0, 0)), "exceeds the") - }) - - t.Run("a window too thin to satisfy minSamples is a rejection, not a config error", func(t *testing.T) { - t.Parallel() - // Whether the requested depth can ever supply minSamples is a static - // property checked at configuration time. At runtime a thin window is a - // data condition, and must come back with the measured statistics. - shallow := twapSeries(t, []int64{100}) - _, err := twapFunc(anchorNs)(shallow, twapConfigMap(3, 3, 0, 2, 2)) - require.ErrorIs(t, err, ErrTWAPRejected) - var rejection *TWAPRejection - require.ErrorAs(t, err, &rejection) - assert.Equal(t, []TWAPRejectionReason{ReasonInsufficientSamples}, rejection.Reasons) - }) - - t.Run("not a configuration map", func(t *testing.T) { - t.Parallel() - require.ErrorIs(t, call(42), ErrTWAPConfig) - }) - - t.Run("not a window", func(t *testing.T) { - t.Parallel() - _, err := twapFunc(anchorNs)(decimal.NewFromInt(1), twapConfigMap(3, 1, 0, 0, 0)) - require.ErrorContains(t, err, "expects a history window") - }) -} - -// TestTWAP_NonPositivePriceRejected covers the log-space requirement: a -// non-positive price has no logarithm, so the window cannot be filled. -func TestTWAP_NonPositivePriceRejected(t *testing.T) { - t.Parallel() - - series, err := NewSeries( - []decimal.Decimal{decimal.NewFromInt(100), decimal.Zero}, - []uint64{twapWindowStartSeconds * uint64(time.Second), (twapWindowStartSeconds + 1) * uint64(time.Second)}, - ) - require.NoError(t, err) - - anchorNs := uint64(twapWindowStartSeconds+3) * uint64(time.Second) - _, err = twapFunc(anchorNs)(series, twapConfigMap(3, 1, 2, 2, 2)) - require.ErrorContains(t, err, "must be positive") -} - -// TestTWAP_Unbound covers the patch-bypass equivalent for TWAP: without a round -// to anchor the window, it must refuse rather than invent one. -func TestTWAP_Unbound(t *testing.T) { - t.Parallel() - - _, err := twapUnbound(nil, nil) - require.ErrorContains(t, err, "no observation timestamp bound") - - // A pooled environment always carries a bound TWAP; release restores the - // unbound default. - env := NewEnv(uint64(5 * time.Second)) - bound, ok := env["TWAP"].(func(any, any) (decimal.Decimal, error)) - require.True(t, ok, "NewEnv must bind TWAP to the round") - env.release() - - series := twapSeries(t, []int64{100, 200, 300}) - _, err = bound(series, twapConfigMap(3, 1, 0, 0, 0)) - require.Error(t, err, "the round anchor is 5s, so the window is far from these observations") -} - -// TestTWAP_RejectionMessage checks an operator can tell what failed without -// reproducing the calculation. -func TestTWAP_RejectionMessage(t *testing.T) { - t.Parallel() - - prices := []int64{100, 0, 0, 0, 0} - series := twapSeries(t, prices) - anchorNs := uint64(twapWindowStartSeconds+len(prices)) * uint64(time.Second) - - _, err := twapFunc(anchorNs)(series, twapConfigMap(len(prices), 3, 0, 5, 2)) - require.Error(t, err) - - var rejection *TWAPRejection - require.ErrorAs(t, err, &rejection) - assert.Equal(t, 1, rejection.M) - assert.Equal(t, 4, rejection.Gtail) - assert.Equal(t, 3, rejection.MinSamples) - assert.Equal(t, 2, rejection.MaxTailGap) - - message := err.Error() - for _, want := range []string{"min_samples", "tail_gap_too_long", "M=1/3", "Gtail=4/2"} { - assert.Contains(t, message, want) - } - assert.True(t, errors.Is(err, ErrTWAPRejected)) - assert.Contains(t, fmt.Sprint(err), "rejected") -} diff --git a/llo/protocol/calculated/history_ast.go b/llo/protocol/calculated/history_ast.go index 7de29c1b..40316d6a 100644 --- a/llo/protocol/calculated/history_ast.go +++ b/llo/protocol/calculated/history_ast.go @@ -39,6 +39,11 @@ const HistoryFunctionName = "History" // static analysis that validates its configuration. const twapFunctionName = "TWAP" +// twapMaxWindowSeconds bounds the number of one-second buckets a single TWAP +// evaluation may allocate and fill. 24 hours is far beyond any settlement window +// while keeping the per-round work bounded. +const twapMaxWindowSeconds = 24 * 60 * 60 + // Field selects which part of a stored stream value a window projects. One // stored window serves every field, so History(s1, 10), History(s1_bid, 10) and // History(s1_ask, 10) share a single series in state and differ only here. @@ -304,16 +309,11 @@ func (p *historyPatcher) rewrite(node *ast.Node, call *ast.CallNode) { p.refByNode[*node] = ref } -// checkTWAP validates a TWAP call against the depth of the window it reads. -// -// This is the static half of TWAP validation: whether a configuration can ever be -// satisfied is a property of the expression, so it belongs here rather than at -// evaluation time, where the same condition would surface as a per-round -// rejection and look like a data problem instead of a deployment mistake. +// checkTWAP validates a TWAP call at compile time: arity, per-expression call +// count, and static satisfiability of minSamples against the history depth. // -// Only literal configuration can be checked. A configuration built at runtime is -// left to the runtime validation in functions_twap.go, which is stricter but -// later. +// Only literal configuration can be checked. A configuration built at runtime +// is left to runtime validation, which is stricter but later. func (p *historyPatcher) checkTWAP(call *ast.CallNode) { // Counted first, and counted whatever the call looks like: this is the one // TWAP check that does not depend on the configuration being literal, which @@ -344,9 +344,7 @@ func (p *historyPatcher) checkTWAP(call *ast.CallNode) { } // Compared as int64: minSamples is a literal and can be any integer the // parser accepted, so narrowing it to the width of ref.Count would let a - // value above 2^32 wrap into a small one and pass. The runtime validation - // still rejects it, but the diagnostic this check exists to give would be - // lost. + // value above 2^32 wrap into a small one and pass. if minSamples < 1 { p.errorf("%s requires minSamples to be at least 1, got %d", twapFunctionName, minSamples) return diff --git a/llo/protocol/calculated/process_fuzz_test.go b/llo/protocol/calculated/process_fuzz_test.go index 27e8b523..cfc66e2d 100644 --- a/llo/protocol/calculated/process_fuzz_test.go +++ b/llo/protocol/calculated/process_fuzz_test.go @@ -73,7 +73,6 @@ func FuzzProcessCalculatedStreams(f *testing.F) { "EMA(History(s1, 3), 2)", "SMA(History(s1, 3), 2)", "Stddev(History(s1, 3))", - `TWAP(History(s1, 3), {window: Duration("3s"), minSamples: 1, maxHeadGap: 3, maxInteriorGap: 3, maxTailGap: 3})`, // Deeper than the reader serves: the whole round takes the warmup gate. "Avg(History(s1, 64))", // Two windows in one expression, so binding order is exercised. diff --git a/llo/protocol/calculated/program.go b/llo/protocol/calculated/program.go index d9d46a06..52026387 100644 --- a/llo/protocol/calculated/program.go +++ b/llo/protocol/calculated/program.go @@ -84,8 +84,7 @@ func AnalyzeExpressionHistory(expression string) ([]HistoryRef, error) { // // It is the check to run before a channel definition reaches consensus: it // parses, rewrites History calls, and applies every static rule (argument -// shapes, depth caps, per-expression fan-out, window positions, reserved names, -// TWAP configuration satisfiability). It does not evaluate, so it needs no +// shapes, depth caps, per-expression fan-out, window positions, reserved names). It does not evaluate, so it needs no // stream values and no persisted state, and it is a pure function of the // expression string. // diff --git a/llo/protocol/calculated/series.go b/llo/protocol/calculated/series.go index e5ac10dc..fcd38b7c 100644 --- a/llo/protocol/calculated/series.go +++ b/llo/protocol/calculated/series.go @@ -187,7 +187,7 @@ type HistoryReader interface { type syntheticHistoryReader struct { // endNanoseconds is the exclusive upper bound of the synthesized // timestamps, which must be the round's observation timestamp: functions - // that place records into a window relative to it (TWAP) would otherwise see + // that place records into a window relative to it would otherwise see // every record fall outside the window. endNanoseconds uint64 // intervalNanoseconds is the spacing between synthesized records. diff --git a/llo/protocol/calculated/validation_test.go b/llo/protocol/calculated/validation_test.go index 1e9d94e7..8ea051fa 100644 --- a/llo/protocol/calculated/validation_test.go +++ b/llo/protocol/calculated/validation_test.go @@ -22,9 +22,9 @@ func TestValidateExpression(t *testing.T) { "Count(History(s1, 10))", "Avg(History(s1_bid, 300))", "Div(Avg(History(s1, 10)), s2)", - "EMA(History(s1, 50), 20)", - `TWAP(History(s1, 600), {window: Duration("5m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`, - } { + "EMA(History(s1, 50), 20)", + `TWAP(History(s1, 600), {window: Duration("5m"), minSamples: 240})`, + } { assert.NoError(t, ValidateExpression(expression), "expression %q", expression) } }) @@ -46,49 +46,6 @@ func TestValidateExpression(t *testing.T) { }) } -// TestValidateExpression_TWAPSatisfiability covers the static half of TWAP -// validation: a configuration that can never be satisfied is a deployment -// mistake, and saying so here beats letting every round reject the window and -// look like a data problem. -func TestValidateExpression_TWAPSatisfiability(t *testing.T) { - t.Parallel() - - // 600 records can supply 240 observations. - require.NoError(t, ValidateExpression( - `TWAP(History(s1, 600), {window: Duration("5m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`)) - - // 100 records can never supply 240. - err := ValidateExpression( - `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`) - require.ErrorIs(t, err, ErrHistoryExpression) - assert.Contains(t, err.Error(), "only keeps 100 records") - - // Exactly enough is fine. - require.NoError(t, ValidateExpression( - `TWAP(History(s1, 240), {window: Duration("4m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`)) - - // A minSamples above the width of the record count must not wrap into a - // small value and pass. 2^32+5 would narrow to 5. - err = ValidateExpression( - `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 4294967301, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`) - require.ErrorIs(t, err, ErrHistoryExpression) - assert.Contains(t, err.Error(), "only keeps 100 records") - - // A non-positive minSamples is reported as such rather than as a depth - // problem, which would read as "requires at least 0 observations". - err = ValidateExpression( - `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 0, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`) - require.ErrorIs(t, err, ErrHistoryExpression) - assert.Contains(t, err.Error(), "requires minSamples to be at least 1") - - // A non-literal configuration cannot be checked statically; runtime - // validation still applies. - require.NoError(t, ValidateExpression("TWAP(History(s1, 10), cfg)")) - - // Wrong arity is caught. - require.Error(t, ValidateExpression("TWAP(History(s1, 10))")) -} - func TestValidateChannelExpressions(t *testing.T) { t.Parallel() @@ -170,18 +127,47 @@ func TestValidateChannelExpressions(t *testing.T) { assert.Contains(t, err.Error(), "abi index: 1") } -// TestProcessCalculatedStreamsDryRun_Satisfiability checks the offline path -// rejects the same configurations the static analysis does. -func TestProcessCalculatedStreamsDryRun_Satisfiability(t *testing.T) { +// TestValidateExpression_TWAPSatisfiability covers the static half of TWAP +// validation: a configuration that can never be satisfied is a deployment +// mistake, and saying so here beats letting every round reject the window and +// look like a data problem. +func TestValidateExpression_TWAPSatisfiability(t *testing.T) { t.Parallel() - require.NoError(t, ProcessCalculatedStreamsDryRun( - `TWAP(History(s1, 300), {window: Duration("5m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`)) + // 600 records can supply 240 observations. + require.NoError(t, ValidateExpression( + `TWAP(History(s1, 600), {window: Duration("5m"), minSamples: 240})`)) - err := ProcessCalculatedStreamsDryRun( - `TWAP(History(s1, 10), {window: Duration("5m"), minSamples: 240, maxHeadGap: 30, maxInteriorGap: 10, maxTailGap: 30})`) - require.Error(t, err) - assert.Contains(t, err.Error(), "only keeps 10 records") + // 100 records can never supply 240. + err := ValidateExpression( + `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 240})`) + require.ErrorIs(t, err, ErrHistoryExpression) + assert.Contains(t, err.Error(), "only keeps 100 records") + + // Exactly enough is fine. + require.NoError(t, ValidateExpression( + `TWAP(History(s1, 240), {window: Duration("4m"), minSamples: 240})`)) + + // A minSamples above the width of the record count must not wrap into a + // small value and pass. 2^32+5 would narrow to 5. + err = ValidateExpression( + `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 4294967301})`) + require.ErrorIs(t, err, ErrHistoryExpression) + assert.Contains(t, err.Error(), "only keeps 100 records") + + // A non-positive minSamples is reported as such rather than as a depth + // problem, which would read as "requires at least 0 observations". + err = ValidateExpression( + `TWAP(History(s1, 100), {window: Duration("5m"), minSamples: 0})`) + require.ErrorIs(t, err, ErrHistoryExpression) + assert.Contains(t, err.Error(), "requires minSamples to be at least 1") + + // A non-literal configuration cannot be checked statically; runtime + // validation still applies. + require.NoError(t, ValidateExpression("TWAP(History(s1, 10), cfg)")) + + // Wrong arity is caught. + require.Error(t, ValidateExpression("TWAP(History(s1, 10))")) } // TestValidateExpression_TWAPCallCount covers the bucket bound: each TWAP call @@ -213,7 +199,7 @@ func twapCalls(count int) string { calls := make([]string, 0, count) for i := range count { calls = append(calls, fmt.Sprintf( - `TWAP(History(s%d, 1), {window: Duration("1s"), minSamples: 1, maxHeadGap: 1, maxInteriorGap: 1, maxTailGap: 1})`, + `TWAP(History(s%d, 1), {window: Duration("1s"), minSamples: 1})`, i+1)) } return sumExpressions(calls) @@ -226,3 +212,16 @@ func sumExpressions(expressions []string) string { } return summed } + +// TestProcessCalculatedStreamsDryRun_Satisfiability checks the offline path +// rejects the same configurations the static analysis does. The satisfiable +// case is not tested here because the TWAP stub returns an error at evaluation; +// only the static rejection path is exercised. +func TestProcessCalculatedStreamsDryRun_Satisfiability(t *testing.T) { + t.Parallel() + + err := ProcessCalculatedStreamsDryRun( + `TWAP(History(s1, 10), {window: Duration("5m"), minSamples: 240})`) + require.Error(t, err) + assert.Contains(t, err.Error(), "only keeps 10 records") +} diff --git a/llo/protocol/limits.go b/llo/protocol/limits.go index 33e40e83..3fdf878e 100644 --- a/llo/protocol/limits.go +++ b/llo/protocol/limits.go @@ -83,26 +83,12 @@ const ( MaxHistoryRecordsPerExpression = 4 * MaxHistoryRecordsPerPair // MaxTWAPCallsPerExpression bounds how many TWAP calls a single expression - // may make. - // - // History depth is not a bound on TWAP work. A TWAP call needs only a - // depth-1 history window, so MaxHistoryRecordsPerExpression permits - // thousands of calls in one expression, and each is free to request the - // longest window allowed — one-second buckets allocated and filled every - // round, per channel, on the consensus path. - // - // Capping the count rather than pricing each call by its window is - // deliberate: the count is syntactic, so it holds for a configuration built - // at runtime, which a window-based budget could only bound by inspecting - // literals. It is the coarser limit — every call is charged its worst case — - // and that is the right trade for a limit consensus depends on. - // - // Four allows the shapes that need more than one window (a cross rate, a - // spread between two TWAPs) while holding the per-round ceiling to four - // maximum-length windows. Consensus-relevant, like every limit here: every - // oracle must reject the same expression, so it is never per-node - // configurable. + // may make. Each call may request a maximum-length window of one-second + // buckets allocated and filled every round, so the count is what bounds + // the work. Consensus-relevant: every oracle must reject the same + // expression, so it is never per-node configurable. MaxTWAPCallsPerExpression = 4 + // MaxHistoryRecordBytes is the maximum serialized size of one history // record, enforced on append (StreamHistory.Append) and used as the // per-record size when admitting pairs against MaxHistoryTotalBytes.