Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ scripting-lua = ["dep:mlua"]
# .cargo/config.toml); locally it uses your gh/git credential helper, and in CI the reusable workflow
# rewrites this https URL with the EDGECOMMONS_READ_TOKEN PAT. For local development against an
# unpushed change, temporarily swap this for `path = "../core-monorepo/libs/rust"`.
edgecommons = { git = "https://github.com/edgecommons/edgecommons.git", rev = "1c01ace09321d8f0e052e63ffb7c0a11aceafb94", default-features = false }
edgecommons = { git = "https://github.com/edgecommons/edgecommons.git", rev = "36a70c48b65b35f77bfab70d3a73869debdfc407", default-features = false }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal", "time", "sync"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
Expand Down
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,11 @@ edgecommons monorepo for the full design.

## Unified Namespace (UNS)

The processor speaks the edgecommons **Unified Namespace** — all topics are
`ecv1/{device}/{component}/{instance}/{class}[/channel]` and it appears on the bus as
`ecv1/{device}/telemetry-processor/main/…` (its component token is the short name after the last
`.`). What this means for the processor:
The processor speaks the edgecommons **Unified Namespace** — topics are
`ecv1/{device}/{component}/[{instance}/]{class}[/channel]` (the instance segment is optional), and
because it is a single instance the processor appears on the bus at **component scope**:
`ecv1/{device}/telemetry-processor/{class}[/channel]` — the component token (the short name after the
last `.`) is followed directly by the class, with no instance segment. What this means for the processor:

- **Ingest** the fleet's southbound telemetry (the `data` class) with a single wildcard:
`ecv1/+/+/+/data/#` (or scope it, e.g. `ecv1/+/opcua-adapter/+/data/#`).
Expand Down Expand Up @@ -58,9 +59,9 @@ file sinks + CloudWatch are all on by default, so the command above needs no `--

Publish synthetic `SouthboundSignalUpdate` messages (envelopes with a top-level `identity`) to an
adapter's UNS data topic, e.g. `ecv1/gw-01/opcua-adapter/kep1/data/<signal>`, and watch: downsampled
messages on `ecv1/my-thing/telemetry-processor/main/data/downsampled` (MQTTX), and rolling Parquet
messages on `ecv1/my-thing/telemetry-processor/data/downsampled` (MQTTX), and rolling Parquet
files under `./out/archive/dt=…/`. Subscribe `ecv1/+/+/+/state` to see the processor's automatic
keepalive, and address `ecv1/my-thing/telemetry-processor/main/cmd/get-stats` to read its counters.
keepalive, and address `ecv1/my-thing/telemetry-processor/cmd/get-stats` to read its counters.

## Build the device artifact (Greengrass, Linux)

Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ northbound to MQTT / AWS IoT Core, or append to a durable stream that lands in *
rolling Parquet/AVRO files**. Built on the `edgecommons` library, it is the high-throughput seam between
southbound protocol adapters and the cloud, and runs wherever you deploy it — as a Greengrass v2
component, a standalone process, or a Kubernetes pod. It appears on the bus as
`ecv1/{device}/telemetry-processor/main/…` and answers UNS commands (`get-stats` / `flush` / `pause` /
`ecv1/{device}/telemetry-processor/…` and answers UNS commands (`get-stats` / `flush` / `pause` /
`resume` + the library built-ins).

| Doc | Start here when you want to… |
Expand Down
8 changes: 4 additions & 4 deletions docs/how-to-guides.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ credentials the SDK chain can find (env / profile / instance role).
"subscribe": ["ecv1/+/+/+/data/#"],
"pipeline": [ { "filter": { "field": "body.samples[].quality", "op": "ne", "value": "GOOD" } } ],
"target": "northbound",
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/evt/alarms", "qos": "atLeastOnce" } }
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/evt/alarms", "qos": "atLeastOnce" } }
```

- `target: "northbound"` publishes via IoT Core / the northbound MQTT broker.
Expand All @@ -375,7 +375,7 @@ once and fans every message out to every route that registered it, so the routes
"instances": [
{ "id": "downsample-local", "subscribe": ["ecv1/+/+/+/data/#"],
"pipeline": [ { "filter": { "quality": "GOOD" } }, { "sample": { "everyMs": 1000 } } ],
"target": "local", "publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/data/downsampled" } },
"target": "local", "publish": { "topic": "ecv1/{ThingName}/telemetry-processor/data/downsampled" } },
{ "id": "archive", "subscribe": ["ecv1/+/+/+/data/#"],
"pipeline": [ { "aggregate": { "window": "10s", "by": "body.signal.id", "fn": ["avg", "max"] } } ],
"target": "stream:archive" }
Expand Down Expand Up @@ -427,7 +427,7 @@ filter to specific adapters instead when you don't need the whole fleet, e.g.

**Goal:** inspect and control a running processor from the console / any MQTT client.

The processor answers its command inbox at `ecv1/{device}/telemetry-processor/main/cmd/<verb>`. Send a
The processor answers its command inbox at `ecv1/{device}/telemetry-processor/cmd/<verb>`. Send a
`cmd` envelope (`header.name` = the verb) with `header.reply_to` set to get a structured reply.

| Verb | What it does |
Expand All @@ -440,7 +440,7 @@ The processor answers its command inbox at `ecv1/{device}/telemetry-processor/ma
| `pause` / `resume` | stop / restart enqueuing to a route (`{route}`) or all routes (body omitted) |

The processor also publishes, without any request: its `state` keepalive
(`ecv1/{device}/telemetry-processor/main/state`), a `metric/pipeline` throughput metric (when
(`ecv1/{device}/telemetry-processor/state`), a `metric/pipeline` throughput metric (when
`metricEmission.target: "messaging"`), and `evt/warning/{queue-overflow,route-error,stream-unavailable}`
health events (via the library's `events()` facade). Subscribe the fleet with
`ecv1/+/+/+/{state,metric,evt}/#`.
Expand Down
14 changes: 7 additions & 7 deletions docs/reference/messaging-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The **eight classes**, and how the processor uses each:
| `evt` | application | **Output** — pipeline health events (see [Events](#events-evt)) and forwarded alarms |
| `cmd` | application | **Input** — the library command inbox (built-in + [custom verbs](#command-verbs)) |
| `app` | application | free-form; unused by default |
| `state` | **reserved** (library) | the automatic keepalive on `ecv1/{device}/telemetry-processor/main/state` |
| `state` | **reserved** (library) | the automatic keepalive on `ecv1/{device}/telemetry-processor/state` |
| `metric` | **reserved** (library) | the processor's `metric/pipeline` throughput metric |
| `cfg` | **reserved** (library) | the effective-config publisher |
| `log` | **reserved** (library) | (library-owned) |
Expand Down Expand Up @@ -152,8 +152,8 @@ The output target is per route (`target`). Route outputs must land on a non-rese
| `stream:<name>` | a edgecommons durable stream | partition key from `publish.partitionKey` (default = the route `key`, i.e. `body.signal.id`) | `streams().stream(name).append(record)` |

- Set an explicit `publish.topic` to a UNS `data`/`evt`/`app` topic template, e.g.
`ecv1/{ThingName}/telemetry-processor/main/data/downsampled` or
`ecv1/{ThingName}/telemetry-processor/main/evt/alarms`. Templates are resolved at startup.
`ecv1/{ThingName}/telemetry-processor/data/downsampled` or
`ecv1/{ThingName}/telemetry-processor/evt/alarms`. Templates are resolved at startup.
- **`northbound`** publishes to IoT Core via the mqttproxy with `qos` = `atLeastOnce` (default) or
`atMostOnce`.
- **`stream:<name>`** appends the EdgeCommons protobuf envelope as one record; the stream's
Expand Down Expand Up @@ -205,7 +205,7 @@ except on a `local` target where it is restamped):

## Command verbs

The processor subscribes its own command inbox `ecv1/{device}/telemetry-processor/main/cmd/#` (wired
The processor subscribes its own command inbox `ecv1/{device}/telemetry-processor/cmd/#` (wired
automatically by the library). A `cmd` request whose `header.reply_to` is set gets a structured reply
`{"ok": true, "result": …}` or `{"ok": false, "error": {"code", "message"}}`; a request without
`reply_to` is fire-and-forget.
Expand Down Expand Up @@ -234,8 +234,8 @@ automatically by the library). A `cmd` request whose `header.reply_to` is set ge
## Events (`evt`)

The processor publishes rate-limited health events through the `edgecommons` `events()` publish facade
(`gg.events()`, bound to the `main` instance) on
`ecv1/{device}/telemetry-processor/main/evt/{severity}/{type}` (a non-reserved class; subscribe the
(`gg.events()`, at component scope — no instance token) on
`ecv1/{device}/telemetry-processor/evt/{severity}/{type}` (a non-reserved class; subscribe the
fleet with `ecv1/+/+/+/evt/#`). The facade derives the channel from the body's own `severity`/`type`
(so topic and body can never disagree) and stamps `timestamp`; every one of the processor's own health
events currently uses `severity: "warning"`:
Expand All @@ -255,7 +255,7 @@ forwarding an already-shaped signal reading, not minting a processor health even
## Metrics (`metric`)

With `metricEmission.target: "messaging"` the processor emits a `pipeline` metric every 30 s on
`ecv1/{device}/telemetry-processor/main/metric/pipeline` (subscribe `ecv1/+/+/+/metric/#`), carrying
`ecv1/{device}/telemetry-processor/metric/pipeline` (subscribe `ecv1/+/+/+/metric/#`), carrying
the summed per-interval deltas of the route counters: `messagesIn`, `messagesOut`, `messagesDropped`,
`streamAppends`, `publishFailures`. Per-route detail is available on demand via the `get-stats`
command. System measures (CPU/memory/…) are emitted automatically by the heartbeat as the `sys`
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The telemetry-processor emits pipeline throughput metrics through the EdgeCommon
`metricEmission.target: messaging`, metrics are published on the reserved UNS `metric` class:

```text
ecv1/{device}/telemetry-processor/main/metric/pipeline
ecv1/{device}/telemetry-processor/metric/pipeline
```

The processor does not publish directly to reserved `metric` topics. It defines and emits the metric
Expand Down Expand Up @@ -47,7 +47,7 @@ Dimensions: runtime-injected component dimensions only.
Use the `get-stats` command for per-route counters:

```text
ecv1/{device}/telemetry-processor/main/cmd/get-stats
ecv1/{device}/telemetry-processor/cmd/get-stats
```

The command returns route ids, cumulative in/out/drop counts, stream appends, publish failures,
Expand Down
12 changes: 6 additions & 6 deletions docs/sample-configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ On HOST the dual-MQTT transport needs broker details. You can supply them inline
{ "sample": { "everyMs": 1000, "by": "body.signal.id" } }
],
"target": "local",
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/data/downsampled" }
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/data/downsampled" }
}
]
}
Expand Down Expand Up @@ -459,7 +459,7 @@ publishes them to IoT Core at a chosen QoS.
{ "filter": { "field": "body.samples[].quality", "op": "ne", "value": "GOOD" } }
],
"target": "northbound",
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/evt/alarms", "qos": "atLeastOnce" }
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/evt/alarms", "qos": "atLeastOnce" }
}
]
}
Expand All @@ -472,7 +472,7 @@ publishes them to IoT Core at a chosen QoS.
| `subscribe` | Subscribes the **full `data` class** (`ecv1/+/+/+/data/#`) — there is no `alarms` topic class (the eight are `data`/`evt`/`cmd`/`app`/`state`/`metric`/`cfg`/`log`). It is the `quality != GOOD` filter below (not the subscription) that selects the alarm/fault readings, so only those reach IoT Core — priced per message — keeping northbound sparse. |
| `filter.field` / `op: "ne"` / `value: "GOOD"` | Keep a message when **any** `body.samples[].quality` is not `GOOD` — i.e. a fault/alarm/uncertain reading. (Equivalently a Rhai predicate like `samples.any(\|s\| s.quality != "GOOD")` — see §9.) |
| `target: "northbound"` | Publishes via `publish_northbound` instead of the local bus. |
| `publish.topic` | The IoT Core topic (template-resolved). `ecv1/{ThingName}/telemetry-processor/main/evt/alarms` publishes alarms as the UNS **`evt`** class, namespaced per device (subscribe `ecv1/+/+/+/evt/#`). |
| `publish.topic` | The IoT Core topic (template-resolved). `ecv1/{ThingName}/telemetry-processor/evt/alarms` publishes alarms as the UNS **`evt`** class, namespaced per device (subscribe `ecv1/+/+/+/evt/#`). |
| `publish.qos` | `atLeastOnce` (default) guarantees delivery with possible duplicates; `atMostOnce` is fire-and-forget (cheaper, may drop). Only these two are accepted; anything else falls back to `atLeastOnce`. |

---
Expand Down Expand Up @@ -551,7 +551,7 @@ ComponentConfiguration:
pipeline:
- filter: { field: "body.samples[].quality", op: "ne", value: "GOOD" }
target: "northbound"
publish: { topic: "ecv1/{ThingName}/telemetry-processor/main/evt/alarms", qos: "atLeastOnce" }
publish: { topic: "ecv1/{ThingName}/telemetry-processor/evt/alarms", qos: "atLeastOnce" }
```

The component also needs `ComponentDependencies` and `accessControl` (abbreviated here):
Expand Down Expand Up @@ -685,7 +685,7 @@ processed independently by each route's pipeline and target.
{ "sample": { "everyMs": 1000, "by": "body.signal.id" } }
],
"target": "local",
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/data/downsampled" }
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/data/downsampled" }
},
{
"id": "archive-good",
Expand Down Expand Up @@ -736,7 +736,7 @@ the convenience bindings `value`/`quality` (the first sample's). The example bel
{ "script": "#{ \"signal\": body.signal, \"scaled\": value * 0.1, \"q\": quality, \"src\": topic }" }
],
"target": "local",
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/main/data/scaled" }
"publish": { "topic": "ecv1/{ThingName}/telemetry-processor/data/scaled" }
}
]
}
Expand Down
14 changes: 7 additions & 7 deletions docs/tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ In this tutorial you bring the `telemetry-processor` up on your laptop, feed it
of Python.

By the end you will have seen a downsampled message land on the Unified-Namespace `data` topic
`ecv1/my-thing/telemetry-processor/main/data/downsampled`, and a rolling `.parquet` file appear under
`ecv1/my-thing/telemetry-processor/data/downsampled`, and a rolling `.parquet` file appear under
`./out/archive/dt=…/hr=…/`.

> This is a guided first run — it makes the choices for you and keeps the explanation short. For the
Expand Down Expand Up @@ -54,7 +54,7 @@ subscribed to the fleet's UNS `data` class `ecv1/+/+/+/data/#`:

- **`downsample-local`** — drops any update that isn't all-`GOOD` quality, then keeps **at most one
message per signal per second** (`sample everyMs:1000`), and republishes the survivors on
`ecv1/my-thing/telemetry-processor/main/data/downsampled`. Target `local` (straight back onto the
`ecv1/my-thing/telemetry-processor/data/downsampled`. Target `local` (straight back onto the
bus; the output's `identity` is restamped to the processor so it can't self-echo through the fleet
filter).
- **`archive`** — also drops non-`GOOD`, then rolls each signal's values into **5-second tumbling
Expand Down Expand Up @@ -136,10 +136,10 @@ appears**: the `filter { quality: GOOD }` stage dropped it before sampling. You'
(exact values and cadence depend on arrival timing):

```
ecv1/my-thing/telemetry-processor/main/data/downsampled ns=3;i=1001 = 20.0 [GOOD]
ecv1/my-thing/telemetry-processor/main/data/downsampled ns=3;i=1002 = 1.0 [GOOD]
ecv1/my-thing/telemetry-processor/main/data/downsampled ns=3;i=1001 = 20.8 [GOOD]
ecv1/my-thing/telemetry-processor/main/data/downsampled ns=3;i=1002 = 1.08 [GOOD]
ecv1/my-thing/telemetry-processor/data/downsampled ns=3;i=1001 = 20.0 [GOOD]
ecv1/my-thing/telemetry-processor/data/downsampled ns=3;i=1002 = 1.0 [GOOD]
ecv1/my-thing/telemetry-processor/data/downsampled ns=3;i=1001 = 20.8 [GOOD]
ecv1/my-thing/telemetry-processor/data/downsampled ns=3;i=1002 = 1.08 [GOOD]
```

## 5. Find the Parquet archive
Expand Down Expand Up @@ -192,7 +192,7 @@ results to different channels. That is the whole idea of the processor.
While it ran, the processor was also a full **Unified-Namespace citizen**: subscribe
`ecv1/+/+/+/state` to see its automatic keepalive, `ecv1/+/+/+/metric/#` for its `pipeline` throughput
metric, and `ecv1/+/+/+/evt/#` for health events — and you can address its command inbox at
`ecv1/my-thing/telemetry-processor/main/cmd/get-stats` (or `flush` / `pause` / `resume`, plus the
`ecv1/my-thing/telemetry-processor/cmd/get-stats` (or `flush` / `pause` / `resume`, plus the
library built-ins `ping` / `reload-config` / `get-configuration`) to read the per-route counters. See
the [messaging-interface reference](reference/messaging-interface.md#command-verbs).

Expand Down
2 changes: 1 addition & 1 deletion recipe.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ ComponentConfiguration:
- filter: { field: "body.samples[].quality", op: "ne", value: "GOOD" }
target: "northbound"
publish:
topic: "ecv1/{ThingName}/telemetry-processor/main/evt/alarms"
topic: "ecv1/{ThingName}/telemetry-processor/evt/alarms"
qos: "atLeastOnce"
accessControl:
aws.greengrass.ipc.pubsub:
Expand Down
4 changes: 2 additions & 2 deletions src/app.rs
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,8 @@ impl ProcessorApp {
// The processor's own UNS identity — the self-echo guard's match, and the restamp source.
let own_device = config.identity().device().to_string();
let own_component = config.identity().component().to_string();
// The `evt` health-event publisher — a thin wrapper over the library's `events()` facade,
// bound to the `main` instance (matches the pre-migration `gg.uns()` topic instance).
// The `evt` health-event publisher — a thin wrapper over the library's `events()` facade at
// component scope (D-U28: no instance token, so events land on `.../telemetry-processor/evt/…`).
let evt = EvtEmitter::new(gg.events());

let mut app = Self {
Expand Down
10 changes: 5 additions & 5 deletions src/observe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
//! by the fan-out handler and the route [`crate::proc::route::Dispatcher`], read by the
//! `get-stats` command and the metric emitter.
//! - [`EvtEmitter`] — a rate-limited publisher of the processor's own **`evt`** events, now a thin
//! wrapper over the library's [`edgecommons::facades::EventsFacade`] (`gg.events()`, bound to the
//! `main` instance): the facade owns the `evt/{severity}/{type}` channel derivation, the body
//! wrapper over the library's [`edgecommons::facades::EventsFacade`] (`gg.events()`, at component
//! scope — D-U28, no instance token): the facade owns the `evt/{severity}/{type}` channel derivation, the body
//! contract (`severity`/`type`/`message`/`timestamp`/`context`), and the envelope/identity
//! stamping — this migration is exactly what `docs/platform/DESIGN-class-facades.md` §1.2 calls
//! out as the drift the facade fixes (the old hand-rolled emitter had **no severity segment** at
Expand All @@ -21,7 +21,7 @@
//! - [`spawn_metric_emitter`] — the periodic task that emits the summed counters as the **`metric`**
//! class through `gg.metrics()` (interval deltas), mirroring the `uns-bridge` `RelayCounters →
//! gg.metrics()` pattern. With `metricEmission.target: "messaging"` the messaging metric target
//! lands them on `ecv1/{device}/telemetry-processor/main/metric/pipeline`.
//! lands them on `ecv1/{device}/telemetry-processor/metric/pipeline`.

use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
Expand Down Expand Up @@ -88,8 +88,8 @@ impl RouteStats {
}

/// The processor's rate-limited `evt` publisher — a thin wrapper over the library's
/// [`EventsFacade`] (`gg.events()`, bound to the `main` instance, matching the pre-migration
/// `gg.uns()` topic instance). The facade owns the `evt/{severity}/{type}` channel, the body
/// [`EventsFacade`] (`gg.events()`, at component scope — D-U28, no instance token). The facade owns
/// the `evt/{severity}/{type}` channel, the body
/// contract, and the envelope/identity stamping; this type owns only the processor-specific
/// per-channel **cooldown** gate. Publishing is best-effort: `evt` is a non-reserved class, so the
/// reserved-class guard passes; a failed publish is logged at DEBUG and swallowed (the facade
Expand Down
Loading
Loading