From 8cfb71db25a9516f86610812df22a591199fa704 Mon Sep 17 00:00:00 2001 From: Will Killian <2007799+willkill07@users.noreply.github.com> Date: Tue, 7 Jul 2026 14:45:29 -0400 Subject: [PATCH 1/4] docs: reorganize plugin documentation (#361) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #### Overview Reorganize and expand NeMo Relay plugin documentation, including built-in component configuration and all four plugin authoring paths. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Group built-in component configuration under `Configure Plugins` and add configuration guides for model pricing, discoverable plugins, and component-specific settings. - Organize plugin authoring into language binding, native dynamic, and Rust/Python gRPC worker paths. - Add standalone examples, runtime discovery guidance, lifecycle cleanup, compatibility links, and backward-compatible configuration redirects. - Update navigation and entry-point documentation for the new hierarchy. #### Where should the reviewer start? Start with `docs/build-plugins/about.mdx`, `docs/configure-plugins/about.mdx`, and `fern/docs.yml`. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Closes RELAY-317 - Closes RELAY-320 #### Validation - `just docs` - `just docs-linkcheck` - `uv run pre-commit run lychee` - Python, Node.js, and Rust documentation-fence syntax checks ## Summary by CodeRabbit * **Documentation** * Expanded and reorganized plugin documentation: Observability (ATOF/ATIF/exporters), Adaptive, NeMo Guardrails, PII redaction, and Model pricing. * Added new guides for discoverable plugins and dynamic plugin authorship, including Native Dynamic and gRPC Worker (protocol + Python/Rust examples). * Updated lifecycle, validation/initialization/teardown walkthroughs, and clarified semantics around exporters and pricing; removed an older “code examples” page. * **Chores** * Refreshed documentation navigation and standardized internal link paths, including new redirects to preserve older URLs. Authors: - Will Killian (https://github.com/willkill07) Approvers: - Bryan Bednarski (https://github.com/bbednarski9) - https://github.com/lvojtku URL: https://github.com/NVIDIA/NeMo-Relay/pull/361 --- README.md | 35 +- docs/about-nemo-relay/architecture.mdx | 2 +- docs/about-nemo-relay/concepts/plugins.mdx | 75 +++- .../about-nemo-relay/concepts/subscribers.mdx | 14 +- docs/about-nemo-relay/ecosystem.mdx | 4 +- docs/about-nemo-relay/overview.mdx | 16 +- docs/about-nemo-relay/release-notes/index.mdx | 10 +- docs/build-plugins/about.mdx | 73 ++-- docs/build-plugins/code-examples.mdx | 187 --------- docs/build-plugins/dynamic-plugins/about.mdx | 106 +++++ .../dynamic-plugins/grpc-worker/about.mdx | 88 ++++ .../grpc-worker/grpc-worker-protocol.mdx | 135 +++++++ .../grpc-worker/python/about.mdx | 304 ++++++++++++++ .../grpc-worker/rust/about.mdx | 141 +++++++ .../dynamic-plugins/native-dynamic/about.mdx | 157 ++++++++ .../rust-native-plugin-example.mdx | 133 ++++++ .../about.mdx} | 67 +-- .../advanced-configuration.mdx | 46 ++- .../language-binding/code-examples.mdx | 363 +++++++++++++++++ .../language-binding/register-behavior.mdx | 380 ++++++++++++++++++ .../validate-configuration.mdx | 64 ++- docs/build-plugins/register-behavior.mdx | 289 ------------- docs/configure-plugins/about.mdx | 52 +++ .../adaptive}/about.mdx | 10 +- .../adaptive}/acg.mdx | 77 +++- .../adaptive}/adaptive-hints.mdx | 81 +++- .../adaptive}/configuration.mdx | 58 ++- .../discoverable-plugins.mdx | 118 ++++++ docs/configure-plugins/model-pricing.mdx | 94 +++++ .../nemo-guardrails}/about.mdx | 10 +- .../nemo-guardrails}/configuration.mdx | 59 ++- .../observability}/about.mdx | 43 +- .../observability}/atif.mdx | 203 ++++++---- .../observability}/atof.mdx | 114 ++++-- .../observability}/configuration.mdx | 72 +++- .../observability}/openinference.mdx | 106 +++-- .../observability}/opentelemetry.mdx | 89 ++-- .../pii-redaction}/about.mdx | 19 +- .../pii-redaction}/configuration.mdx | 85 ++-- .../plugin-configuration-files.mdx | 163 +++++--- docs/contribute/runtime-contract-docs.mdx | 14 +- docs/getting-started/agent-runtime-primer.mdx | 6 +- docs/getting-started/configuration.mdx | 18 +- docs/getting-started/installation.mdx | 2 +- docs/getting-started/prerequisites.mdx | 4 +- docs/getting-started/quick-start/index.mdx | 16 +- docs/index.yml | 66 ++- .../instrument-llm-call.mdx | 4 +- .../instrument-tool-call.mdx | 4 +- .../provider-response-codecs.mdx | 44 +- .../wrap-tool-calls.mdx | 2 +- docs/nemo-relay-cli/basic-usage.mdx | 2 +- docs/nemo-relay-cli/claude-code.mdx | 2 +- docs/nemo-relay-cli/hermes.mdx | 2 +- docs/nemo-relay-cli/plugin-installation.mdx | 57 ++- .../llm-request-intercept-outcomes.mdx | 6 +- .../tool-execution-intercept-outcomes.mdx | 9 +- docs/resources/support-and-faqs.mdx | 28 +- docs/resources/troubleshooting/index.mdx | 16 +- .../trace-incident-runbook.mdx | 14 +- docs/supported-integrations/deepagents.mdx | 6 +- docs/supported-integrations/langchain.mdx | 6 +- docs/supported-integrations/langgraph.mdx | 6 +- .../openclaw-plugin.mdx | 8 +- fern/docs.yml | 30 ++ 65 files changed, 3338 insertions(+), 1176 deletions(-) delete mode 100644 docs/build-plugins/code-examples.mdx create mode 100644 docs/build-plugins/dynamic-plugins/about.mdx create mode 100644 docs/build-plugins/dynamic-plugins/grpc-worker/about.mdx create mode 100644 docs/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol.mdx create mode 100644 docs/build-plugins/dynamic-plugins/grpc-worker/python/about.mdx create mode 100644 docs/build-plugins/dynamic-plugins/grpc-worker/rust/about.mdx create mode 100644 docs/build-plugins/dynamic-plugins/native-dynamic/about.mdx create mode 100644 docs/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example.mdx rename docs/build-plugins/{basic-guide.mdx => language-binding/about.mdx} (61%) rename docs/build-plugins/{ => language-binding}/advanced-configuration.mdx (71%) create mode 100644 docs/build-plugins/language-binding/code-examples.mdx create mode 100644 docs/build-plugins/language-binding/register-behavior.mdx rename docs/build-plugins/{ => language-binding}/validate-configuration.mdx (66%) delete mode 100644 docs/build-plugins/register-behavior.mdx create mode 100644 docs/configure-plugins/about.mdx rename docs/{adaptive-plugin => configure-plugins/adaptive}/about.mdx (79%) rename docs/{adaptive-plugin => configure-plugins/adaptive}/acg.mdx (79%) rename docs/{adaptive-plugin => configure-plugins/adaptive}/adaptive-hints.mdx (77%) rename docs/{adaptive-plugin => configure-plugins/adaptive}/configuration.mdx (84%) create mode 100644 docs/configure-plugins/discoverable-plugins.mdx create mode 100644 docs/configure-plugins/model-pricing.mdx rename docs/{nemo-guardrails-plugin => configure-plugins/nemo-guardrails}/about.mdx (95%) rename docs/{nemo-guardrails-plugin => configure-plugins/nemo-guardrails}/configuration.mdx (81%) rename docs/{observability-plugin => configure-plugins/observability}/about.mdx (76%) rename docs/{observability-plugin => configure-plugins/observability}/atif.mdx (66%) rename docs/{observability-plugin => configure-plugins/observability}/atof.mdx (73%) rename docs/{observability-plugin => configure-plugins/observability}/configuration.mdx (86%) rename docs/{observability-plugin => configure-plugins/observability}/openinference.mdx (75%) rename docs/{observability-plugin => configure-plugins/observability}/opentelemetry.mdx (78%) rename docs/{pii-redaction-plugin => configure-plugins/pii-redaction}/about.mdx (83%) rename docs/{pii-redaction-plugin => configure-plugins/pii-redaction}/configuration.mdx (86%) rename docs/{build-plugins => configure-plugins}/plugin-configuration-files.mdx (60%) diff --git a/README.md b/README.md index e6d466fa5..46ab6a385 100644 --- a/README.md +++ b/README.md @@ -16,16 +16,16 @@ SPDX-License-Identifier: Apache-2.0 # NVIDIA NeMo Relay -NVIDIA NeMo Relay helps see and control what happens inside agent runs -without rewriting the agent stack already made. It gives coding agents, +NVIDIA NeMo Relay provides visibility into and control over agent runs without +requiring changes to the existing agent stack. It gives coding agents, applications, framework integrations, middleware, and observability backends a shared runtime for scopes, policy, plugins, and lifecycle events. ## Where To Start -| Goal | Start With... | +| Goal | Start With | |---|---| -| Observe Codex, Claude Code, or Hermes locally via CLI | [Quick Start CLI](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/about) | +| Observe Codex, Claude Code, or Hermes locally with the CLI | [Quick Start CLI](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/about) | | Instrument app-owned LLM or tool calls | [Quick Start Application](https://docs.nvidia.com/nemo/relay/getting-started/quick-start) | | Use LangChain, LangGraph, Deep Agents, or OpenClaw | [Supported Integrations](https://docs.nvidia.com/nemo/relay/supported-integrations/about) | | Build a framework or provider integration | [Integrate into Frameworks](https://docs.nvidia.com/nemo/relay/integrate-into-frameworks/about) | @@ -36,9 +36,8 @@ shared runtime for scopes, policy, plugins, and lifecycle events. ## Quick Start CLI -A good first step is to record a real agent run on disk. Once Relay is writing raw -events and a trajectory file, there is something concrete to inspect, debug, and -build from. +Start by recording a real agent run on disk. After Relay writes raw events and a +trajectory file, you have concrete data to inspect, debug, and build on. ### Local Agent Trajectory @@ -50,6 +49,8 @@ trajectories. #### 1. Install the CLI +Run the installer for your platform: + ```bash curl -fsSL https://raw.githubusercontent.com/NVIDIA/NeMo-Relay/main/install.sh | sh ``` @@ -65,7 +66,7 @@ nemo-relay --version ``` The installer supports Linux x86_64/ARM64, macOS Apple Silicon, and Windows -x86_64/ARM64. See the [installation guide](https://docs.nvidia.com/nemo/relay/getting-started/installation) +x86_64/ARM64. Refer to the [installation guide](https://docs.nvidia.com/nemo/relay/getting-started/installation) for version pinning, custom directories, and source-based installation. #### 2. Enable Local Observability Output @@ -82,13 +83,13 @@ The editor creates or updates the nearest project plugin file at then configure these sections: 1. Toggle the Observability component on. -2. Open `ATOF`, toggle the section `[on]` +2. Open `ATOF`. Toggle the section `[on]`. Optionally set: - `output_directory` to `.nemo-relay/atof` - `filename` to `events.jsonl` - `mode` to `overwrite` -3. Open `ATIF`, toggle the section `[on]` +3. Open `ATIF`. Toggle the section `[on]`. Optionally set: - `output_directory` to `.nemo-relay/atif` @@ -97,12 +98,12 @@ then configure these sections: 5. Press `s` to save. > [!NOTE] -> Use `nemo-relay plugins edit` _without_ `--project` only if needing to use these -> exporter settings in a user-level Relay config instead of a specific project. +> Run `nemo-relay plugins edit` without `--project` only when you want +> user-level exporter settings that apply across projects. #### 3. Run Codex or Claude Code Through Relay -Use either host CLI that is installed on a machine. For example: +Run the Relay wrapper for the host CLI installed on your machine. For example: ```bash nemo-relay codex -- exec "Summarize this repository." @@ -119,8 +120,8 @@ and provider settings for that launched process, then shuts the gateway down when the agent exits. > [!WARNING] -> Codex users may need to review and activate generated hooks before events -> appear. Using the Codex Desktop App also adds further complications. +> If generated hooks are inactive, Codex users must review and activate them +> before events appear. The Codex Desktop App has additional limitations. > Refer to the [Codex CLI guide](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/codex) for the > current hook activation caveat and troubleshooting steps. @@ -170,7 +171,7 @@ A successful run creates several outputs to inspect: #### Next Steps -Go to the full [NeMo Relay CLI](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/about) docs for +Refer to the full [NeMo Relay CLI](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/about) docs for persistent host plugin installation, gateway configuration, exporter options, and agent-specific diagnostics. @@ -211,7 +212,7 @@ Then run a minimal example workflow for that binding: ## What Relay Adds -Relay is the liaison between agent systems. A production application may +Relay connects agent systems. A production application can combine NeMo Agent Toolkit, LangChain, LangGraph, provider SDKs, custom harness code, NeMo Guardrails, tracing systems, and evaluation pipelines. Relay gives those pieces one runtime contract instead of asking every layer to invent its diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index f84173559..155c3aeef 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -167,7 +167,7 @@ Two distinctions matter: - Intercepts affect the real execution path - Sanitize guardrails affect the emitted observability payload -For the expanded request-to-response runtime path, including streaming and subscriber handoff, see [Middleware](/about-nemo-relay/concepts/middleware#detailed-execution-flow). +For the expanded request-to-response runtime path, including streaming and subscriber handoff, refer to [Middleware](/about-nemo-relay/concepts/middleware#detailed-execution-flow). ## Runtime Layers diff --git a/docs/about-nemo-relay/concepts/plugins.mdx b/docs/about-nemo-relay/concepts/plugins.mdx index 4ac0ada6b..0f9b0e445 100644 --- a/docs/about-nemo-relay/concepts/plugins.mdx +++ b/docs/about-nemo-relay/concepts/plugins.mdx @@ -1,6 +1,6 @@ --- title: "Plugins" -description: "" +description: "Understand NeMo Relay plugin configuration, lifecycle, ownership, and built-in components." position: 3 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -147,11 +147,16 @@ Plugins install runtime behavior; they do not create a separate execution model. Scopes still own parentage and cleanup, middleware still owns execution ordering, and events still own the canonical runtime record. -## Built-In Plugin Examples +## Built-In Plugin Components -Core plugin APIs register built-in components before lookup, validation, and -initialization. Applications can still register custom plugins, but first-party -components are available by kind without an explicit registration call. +The core runtime registers the `observability`, `nemo_guardrails`, and +`pricing` components before lookup, validation, and initialization. The CLI and +the Python and Node.js bindings also register `adaptive` and `pii_redaction`. +Direct Rust applications must register Adaptive and PII Redaction from their +component crates before validating or initializing either kind: +`nemo_relay_adaptive::plugin_component::register_adaptive_component()` and +`nemo_relay_pii_redaction::component::register_pii_redaction_component()`. +Applications can still register custom plugins. ### Adaptive @@ -159,27 +164,29 @@ Adaptive is implemented as a built-in plugin component. It is not a separate runtime model. It uses the same plugin system as custom components. This matters conceptually because adaptive behavior is configured and activated -through the same component lifecycle as other plugins: +through the same component lifecycle as other plugins. Direct Rust applications +follow this sequence: -- Validate the config -- Initialize the plugin system -- Inspect the activation result if needed +1. Call `nemo_relay_adaptive::plugin_component::register_adaptive_component()`. +2. Validate the config. +3. Initialize the plugin system. +4. Inspect the activation result if needed. Detailed adaptive configuration belongs in -[Adaptive Configuration](/adaptive-plugin/configuration), -[Adaptive Cache Governor (ACG)](/adaptive-plugin/acg), and -[Adaptive Hints](/adaptive-plugin/adaptive-hints). +[Adaptive Configuration](/configure-plugins/adaptive/configuration), +[Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), and +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints). ### Observability The core crate ships a built-in `observability` plugin component for Agent Trajectory Observability Format (ATOF), Agent Trajectory Interchange Format (ATIF), OpenTelemetry, and OpenInference exporters. Each exporter section is -disabled unless its section sets `enabled: true`, and subscriber names are +disabled unless its section sets `enabled = true`, and subscriber names are inferred from the plugin namespace instead of exposed in public config. Detailed observability plugin configuration belongs in -[Observability Configuration](/observability-plugin/configuration). +[Observability Configuration](/configure-plugins/observability/configuration). ### NeMo Guardrails @@ -194,10 +201,44 @@ The current shipped user-facing paths are: subprocess worker Detailed Guardrails plugin configuration belongs in -[NeMo Guardrails Configuration](/nemo-guardrails-plugin/configuration). +[NeMo Guardrails Configuration](/configure-plugins/nemo-guardrails/configuration). -For the CLI gateway's `plugins.toml` discovery, precedence, merge, and editing -rules, see [Plugin Configuration Files](/build-plugins/plugin-configuration-files). +### PII Redaction + +The `pii_redaction` component sanitizes emitted observability payloads without +changing real callback arguments or results. The CLI and primary language +bindings register this component. Direct Rust applications must call +`nemo_relay_pii_redaction::component::register_pii_redaction_component()` +before they validate or initialize a PII Redaction component. + +Configure actions, detectors, targets, and backend modes through [PII Redaction +Configuration](/configure-plugins/pii-redaction/configuration). + +### Model Pricing + +The core crate ships a built-in `pricing` component. It loads catalog sources +that response codecs can use to annotate managed LLM responses with cost +estimates. Configure catalog sources through [Model Pricing](/configure-plugins/model-pricing). + +For `plugins.toml` discovery, precedence, merge, and gateway editing rules, +refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files). + +## Discoverable Plugins + +Discoverable plugins use the same component lifecycle, but the CLI reads a +`relay-plugin.toml` manifest before it creates an internal component. The +manifest identifies a Rust native shared library or a local `grpc-v1` worker, +declares compatibility and capabilities, and supplies integrity evidence for +the artifact. + +The operator keeps the manifest reference and component configuration in +`plugins.toml`. Use `nemo-relay plugins validate ` to check the +manifest, optional static schema, host policy, compatibility, and trust +evidence before enabling or running a dynamic plugin. During startup, Relay +loads the enabled adapter and then validates the synthesized component. Refer +to [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) +for the operator workflow and [Discoverable Plugins](/build-plugins/dynamic-plugins/about) +for the authoring model. ## Practical Guidance diff --git a/docs/about-nemo-relay/concepts/subscribers.mdx b/docs/about-nemo-relay/concepts/subscribers.mdx index 19c6a7cd3..b794a7966 100644 --- a/docs/about-nemo-relay/concepts/subscribers.mdx +++ b/docs/about-nemo-relay/concepts/subscribers.mdx @@ -1,6 +1,6 @@ --- title: "Subscribers" -description: "" +description: "Understand NeMo Relay subscribers, delivery behavior, scope visibility, and exporter roles." position: 5 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -46,7 +46,7 @@ Scope-local subscribers are owned by one active scope and disappear when that scope closes. Deregistering a subscriber affects future emissions. Events that were already -emitted carry a subscriber snapshot, so queued callbacks from that snapshot may +emitted carry a subscriber snapshot, so queued callbacks from that snapshot can still run after deregistration. ### Plugin-Installed Subscribers @@ -117,7 +117,7 @@ flowchart ``` The important boundary is that subscribers do not define the event schema. They -receive the runtime event and may serialize it through the binding helper when +receive the runtime event and can serialize it through the binding helper when they need a stable JSON payload. Exporter subscribers, such as the ATOF JSONL exporter, consume the same event stream and serialize the same canonical event shape for their target backend. @@ -158,13 +158,13 @@ of the canonical event stream. ### Agent Trajectory Interchange Format (ATIF) Exporter -The [Agent Trajectory Interchange Format (ATIF) exporter](/observability-plugin/atif) +The [Agent Trajectory Interchange Format (ATIF) exporter](/configure-plugins/observability/atif) collects lifecycle events and emits trajectory artifacts for offline analysis, replay, or debugging. ### Agent Trajectory Observability Format (ATOF) JSONL Exporter -The [Agent Trajectory Observability Format (ATOF) JSONL exporter](/observability-plugin/atof) +The [Agent Trajectory Observability Format (ATOF) JSONL exporter](/configure-plugins/observability/atof) writes the canonical event stream to a native filesystem path as one raw ATOF event per line. @@ -179,9 +179,9 @@ The OpenInference subscriber maps runtime events into OTLP traces using OpenInference semantics for model-centric observability. Detailed setup, configuration, and API shape for these subscribers belongs in -[Observability](/observability-plugin/about). +[Observability](/configure-plugins/observability/about). For configuration-driven setup, use the built-in -[`observability` plugin](/observability-plugin/configuration) +[`observability` plugin](/configure-plugins/observability/configuration) to install ATOF, ATIF, OpenTelemetry, and OpenInference subscribers from one plugin component. diff --git a/docs/about-nemo-relay/ecosystem.mdx b/docs/about-nemo-relay/ecosystem.mdx index 602b90dcb..61a16ce18 100644 --- a/docs/about-nemo-relay/ecosystem.mdx +++ b/docs/about-nemo-relay/ecosystem.mdx @@ -1,6 +1,6 @@ --- title: "Ecosystem" -description: "" +description: "Understand how NeMo Relay fits with agent frameworks, providers, and the NVIDIA NeMo ecosystem." position: 3 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -115,4 +115,4 @@ Use these links to continue into adjacent concepts and workflows. - [Adding Framework Scopes](/integrate-into-frameworks/adding-scopes) - [Wrapping Tool Calls](/integrate-into-frameworks/wrap-tool-calls) - [Wrapping LLM Calls](/integrate-into-frameworks/wrap-llm-calls) -- [Plugin Model](/build-plugins/basic-guide) +- [Language Binding Plugins](/build-plugins/language-binding/about) diff --git a/docs/about-nemo-relay/overview.mdx b/docs/about-nemo-relay/overview.mdx index 45515f5ad..8262b319b 100644 --- a/docs/about-nemo-relay/overview.mdx +++ b/docs/about-nemo-relay/overview.mdx @@ -1,6 +1,6 @@ --- title: "Overview" -description: "" +description: "Learn how NeMo Relay manages scopes, middleware, plugins, and lifecycle events for agent systems." position: 1 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -14,7 +14,7 @@ applications, framework integrations, middleware, and observability backends a shared runtime for scopes, policy, plugins, and lifecycle events. Agent systems usually cross several boundaries in one request: an entrypoint -starts work, a model is called, tools run, subagents may branch off, and +starts work, a model is called, tools run, subagents can branch off, and observability or policy systems need to understand what happened. Relay gives those boundaries one runtime contract instead of asking each layer to invent its own wrappers, trace vocabulary, and cleanup rules. @@ -46,7 +46,7 @@ Pick the row closest to what you are trying to do. | Instrument application-owned LLM or tool calls | [Instrument Applications](/instrument-applications/about) | Direct SDK instrumentation gives Relay full managed-call semantics around callbacks your code owns. | | Use LangChain, LangGraph, Deep Agents, or OpenClaw | [Supported Integrations](/supported-integrations/about) | Maintained integrations use public framework or plugin APIs where they preserve enough lifecycle fidelity. | | Build a framework, host, or provider integration | [Integrate into Frameworks](/integrate-into-frameworks/about) | Integration guidance helps you choose managed wrappers, explicit lifecycle APIs, hook replay, provider codecs, or upstream support. | -| Package reusable exporters, middleware, or policy | [Build Plugins](/build-plugins/about), [Observability](/observability-plugin/about), and [NeMo Guardrails Plugin](/nemo-guardrails-plugin/about) | Plugins are the configuration-driven path for behavior that should be shared across applications or teams. | +| Package reusable exporters, middleware, or policy | [Build Plugins](/build-plugins/about) and [Configure Plugins](/configure-plugins/about) | Plugins are the configuration-driven path for behavior that should be shared across applications or teams. | | Develop or validate the repository itself | [Development Setup](/contribute/development-setup) and [Testing and Docs](/contribute/testing-and-docs) | Use the contributor workflow when you are changing Relay source, docs, examples, bindings, or integrations. | @@ -57,10 +57,10 @@ guardrails, or adaptive behavior. ## Validate Raw Capture First -Start with [Agent Trajectory Observability Format (ATOF) JSONL](/observability-plugin/atof), +Start with [Agent Trajectory Observability Format (ATOF) JSONL](/configure-plugins/observability/atof), the raw canonical event stream. It shows the lifecycle events Relay actually captured before anything is translated into -[Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif), +[Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif), OpenTelemetry, or OpenInference output. A good first integration process workflow is as follows: @@ -116,9 +116,9 @@ Use the tasks below to build your understanding and set up Relay: |---|---| | Install packages | [Installation](/getting-started/installation) | | Understand the mental model | [Agent Runtime Primer](/getting-started/agent-runtime-primer) | -| Configure plugin files | [Plugin Configuration Files](/build-plugins/plugin-configuration-files) | -| Export traces or trajectories | [Observability](/observability-plugin/about) | -| Tune performance with adaptive behavior | [Adaptive](/adaptive-plugin/about) | +| Configure plugin files | [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) | +| Export traces or trajectories | [Observability](/configure-plugins/observability/about) | +| Tune performance with adaptive behavior | [Adaptive](/configure-plugins/adaptive/about) | | Debug trace incidents | [Trace Incident Runbook](/resources/troubleshooting/trace-incident-runbook) | | Look up symbols | [APIs](/reference/api) | diff --git a/docs/about-nemo-relay/release-notes/index.mdx b/docs/about-nemo-relay/release-notes/index.mdx index 79ab9d7e7..9ae766dbf 100644 --- a/docs/about-nemo-relay/release-notes/index.mdx +++ b/docs/about-nemo-relay/release-notes/index.mdx @@ -1,6 +1,6 @@ --- title: "Release Notes" -description: "" +description: "Read NeMo Relay release notes and find the official release history." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -58,11 +58,11 @@ This release includes: For the major 0.4 additions, start with: - [Plugin Installation](/nemo-relay-cli/plugin-installation) -- [Observability Plugin](/observability-plugin/about) -- [Agent Trajectory Observability Format (ATOF)](/observability-plugin/atof) -- [Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif) +- [Observability Plugin](/configure-plugins/observability/about) +- [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) +- [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) - [Provider Response Codecs And Pricing](/integrate-into-frameworks/provider-response-codecs) -- [NeMo Guardrails Plugin](/nemo-guardrails-plugin/about) +- [NeMo Guardrails Plugin](/configure-plugins/nemo-guardrails/about) - [PII Redaction API Reference](/reference/api/python-library-reference/pii-redaction) - [OpenClaw Plugin Guide](/supported-integrations/openclaw-plugin) - [Hermes CLI Guide](/nemo-relay-cli/hermes) diff --git a/docs/build-plugins/about.mdx b/docs/build-plugins/about.mdx index e32d89046..d4d2a3523 100644 --- a/docs/build-plugins/about.mdx +++ b/docs/build-plugins/about.mdx @@ -1,13 +1,14 @@ --- -title: "About" -description: "" +title: "Build Plugins" +description: "Build code-driven, native dynamic, and gRPC worker plugins for NeMo Relay." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -Use this section when you want to package reusable NeMo Relay behavior as a plugin that can be activated from configuration. +Use this section when you want to create and package reusable NeMo Relay +behavior as a plugin. Plugins are the configuration-driven packaging layer for shared runtime behavior. A plugin can validate component-local config, register middleware and @@ -18,42 +19,60 @@ Plugins prevent repeated registration code for policies, request transforms, exporters, and related runtime components. They give shared behavior a stable kind name, a structured config document, and a clear activation lifecycle. -NeMo Relay also ships built-in plugin components for shared runtime behavior. -For example, the `pricing` component installs model pricing sources that -response codecs can use to annotate managed LLM responses with cost estimates. -Applications, eval harnesses, custom agents, and framework integrations can -activate that component through the same plugin APIs they use for custom -plugins; the `nemo-relay` CLI is only one host that can load plugin -configuration from files. +## When to Use This Guide -## Start Here When - -Use these signals to decide whether this documentation path matches your current task. +Use this guide when you need to package reusable NeMo Relay behavior. - Ship policy bundles across applications -- Install observability exporters consistently -- Install model pricing sources for cost estimates across applications, - harnesses, or agent integrations - Package framework-agnostic request transforms - Validate operator-supplied config before runtime behavior changes +- Package a manifest-backed native library or worker for discovery by the CLI + +Keep behavior scope-local when it applies to only one request or tenant. Use a +process-level plugin only for reusable behavior. + +## Plugin Types + +Choose the group that matches the process boundary and language of the plugin. + +### Language Binding Plugins (Rust, Python, Node.js) + +Application code registers these in-process plugins directly. Start with +[Language Binding Plugins](/build-plugins/language-binding/about) for the +common contract and binding-specific registration examples. + +### Native Dynamic Plugins (Rust) + +These plugins run in-process from a separately packaged Rust shared library. +Start with [Native Dynamic Plugins (Rust)](/build-plugins/dynamic-plugins/native-dynamic/about). + +### gRPC Worker Plugins (Rust) + +These plugins run out of process as a Relay-managed Rust worker. Use [gRPC +Worker Plugins (Rust)](/build-plugins/dynamic-plugins/grpc-worker/rust/about). -If the behavior applies to only one request or tenant, consider scope-local middleware before turning it into a process-level plugin. +### gRPC Worker Plugins (Python) -## Guides +These plugins run out of process as a Relay-managed Python worker. Use [gRPC +Worker Plugins (Python)](/build-plugins/dynamic-plugins/grpc-worker/python/about). -Use these guide links to move from the overview into task-specific instructions. +## Next Steps -- [Define a Plugin](/build-plugins/basic-guide) explains plugin kinds, shape, runtime ownership, and the activation lifecycle. -- [Validate Plugin Configuration](/build-plugins/validate-configuration) covers JSON-compatible config, validation rules, and structured diagnostics. -- [Plugin Configuration Files](/build-plugins/plugin-configuration-files) documents `plugins.toml` file discovery, precedence, merge behavior, and editor controls for the CLI gateway. -- [Register Plugin Behavior](/build-plugins/register-behavior) shows how to initialize config and install subscribers or middleware through `PluginContext`. -- [Design Plugin Configuration](/build-plugins/advanced-configuration) covers validation rules, advanced configuration patterns, rollout controls, and `PluginContext` usage. -- [NeMo Guardrails Plugin](/nemo-guardrails-plugin/about) documents the built-in first-party `nemo_guardrails` component. -- [Code Examples](/build-plugins/code-examples) provides patterns for dynamic header injection, subscriber-oriented export, multi-surface bundles, and framework-facing plugins. +- Build a language binding plugin in this order: + 1. [Validate Plugin Configuration](/build-plugins/language-binding/validate-configuration) + 2. [Register Plugin Behavior](/build-plugins/language-binding/register-behavior) + 3. [Design Plugin Configuration](/build-plugins/language-binding/advanced-configuration) + 4. [Code Examples](/build-plugins/language-binding/code-examples) +- Build a shared-library example with [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example). +- Compare manifest-backed packages in [Discoverable Plugins](/build-plugins/dynamic-plugins/about). +- Choose a worker runtime in [gRPC Worker Plugin Concepts](/build-plugins/dynamic-plugins/grpc-worker/about) + and implement the boundary with [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol). +- Configure a built-in or packaged plugin in [Configure Plugins](/configure-plugins/about). Start by deciding which runtime surfaces the plugin owns: middleware, subscribers, or a combination of related runtime behavior. Define the smallest JSON-compatible config that can drive that behavior, validate it before registration, and keep external objects or callables out of the config document. -Use plugins for reusable process-level behavior. Keep request-specific behavior scope-local so it is cleaned up with the owning scope. +Use plugins for reusable process-level behavior. Keep request-specific behavior +scope-local so Relay removes it when the owning scope closes. diff --git a/docs/build-plugins/code-examples.mdx b/docs/build-plugins/code-examples.mdx deleted file mode 100644 index 068e33f1d..000000000 --- a/docs/build-plugins/code-examples.mdx +++ /dev/null @@ -1,187 +0,0 @@ ---- -title: "Code Examples" -description: "" -position: 8 ---- -{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -SPDX-License-Identifier: Apache-2.0 */} - - -This page collects concrete examples for the surrounding guide area. - -## Dynamic Header Injection - -Use an LLM request intercept when a plugin needs to inject tenant or routing metadata into every provider request. - -LLM request intercepts receive three arguments: `name`, `request`, and `annotated`. The `request` object is immutable, however it is possible to return a new instance of the request with edits, the exception to this is when the intercept is written in Rust. - - - -```python -from typing import Any - -import nemo_relay - -class HeaderPlugin: - def validate(self, plugin_config: dict[str, Any]) -> list[dict[str, str]]: - if "header_name" not in plugin_config or "value" not in plugin_config: - return [{ - "level": "error", - "code": "header-plugin.invalid_config", - "message": "header_name and value are required", - }] - return [] - - def register(self, plugin_config: dict[str, Any], context: nemo_relay.plugin.PluginContext): - def add_header( - name: str, - request: nemo_relay.LLMRequest, - annotated: nemo_relay.AnnotatedLLMRequest | None - ) -> nemo_relay.LLMRequestInterceptOutcome: - # The request object is immutable, however we can return a new instance with updated headers. - headers = request.headers.copy() - headers[plugin_config["header_name"]] = plugin_config["value"] - return nemo_relay.LLMRequestInterceptOutcome( - nemo_relay.LLMRequest(headers=headers, content=request.content), - annotated, - ) - - context.register_llm_request_intercept("inject-header", 100, False, add_header) - -nemo_relay.plugin.register("header-plugin", HeaderPlugin()) -``` - - - -```ts -import * as plugin from 'nemo-relay-node/plugin'; - -const headerPlugin: plugin.Plugin = { - validate(pluginConfig) { - if (typeof pluginConfig.header_name !== 'string' || typeof pluginConfig.value !== 'string') { - return [ - { - level: 'error', - code: 'header-plugin.invalid_config', - message: 'header_name and value are required', - }, - ]; - } - return []; - }, - register(pluginConfig, context) { - context.registerLlmRequestIntercept('inject-header', 100, false, ({ request, annotated }) => ({ - request: { - ...request, - headers: { - ...(request.headers as Record), - [String(pluginConfig.header_name)]: String(pluginConfig.value), - }, - }, - annotated, - })); - }, -}; - -plugin.register('header-plugin', headerPlugin); -``` - - - -This pattern is useful for: - -- Tenant identity -- Trace correlation -- Region or deployment routing - -## OpenInference Export - -Use a subscriber-oriented plugin when the component should watch the full lifecycle rather than rewrite requests. - - - -```python -import nemo_relay - -class OpenInferencePlugin: - def validate(self, plugin_config): - if "endpoint" not in plugin_config: - return [{ - "level": "error", - "code": "openinference-export.invalid_config", - "message": "endpoint is required", - }] - return [] - - def register(self, plugin_config, context): - endpoint = plugin_config["endpoint"] - - def on_event(event): - print("export", endpoint, event.kind, event.name) - - context.register_subscriber("openinference-export", on_event) - -nemo_relay.plugin.register("openinference-export", OpenInferencePlugin()) -``` - - - -```ts -import * as plugin from 'nemo-relay-node/plugin'; - -const openInferencePlugin: plugin.Plugin = { - validate(pluginConfig) { - if (typeof pluginConfig.endpoint !== 'string') { - return [ - { - level: 'error', - code: 'openinference-export.invalid_config', - message: 'endpoint is required', - }, - ]; - } - return []; - }, - register(pluginConfig, context) { - const endpoint = String(pluginConfig.endpoint); - context.registerSubscriber('openinference-export', (event) => { - console.log('export', endpoint, event.kind, event.name); - }); - }, -}; - -plugin.register('openinference-export', openInferencePlugin); -``` - - - -This is the right pattern when the component: - -- Exports traces or metrics -- Aggregates events across tools and LLMs -- Should not change execution behavior - -## Multi-Surface Policy Bundle - -A plugin can register more than one runtime surface when one configuration document controls a related behavior bundle. - -For example, a policy bundle can install: - -- A telemetry subscriber -- LLM request intercepts for request metadata -- Tool guardrails for policy enforcement -- Sanitize guardrails for exported payloads -- Shared component-local state used by those hooks - -Use this pattern when the configured behavior is easier to reason about as one component than as several unrelated plugin components. Keep each registered surface small and make the component config explicit about which surfaces are enabled. - -## Framework-Facing Plugins - -Plugins can stay framework-agnostic if they operate on the normalized runtime data rather than framework-specific objects. - -Good examples: - -- Rewrite provider headers -- Emit tracing data -- Attach scheduling hints -- Apply cross-framework safety policies diff --git a/docs/build-plugins/dynamic-plugins/about.mdx b/docs/build-plugins/dynamic-plugins/about.mdx new file mode 100644 index 000000000..bd3d0bcee --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/about.mdx @@ -0,0 +1,106 @@ +--- +title: "Discoverable Plugins" +description: "Package Rust native libraries and local gRPC workers as discoverable NeMo Relay plugins." +position: 9 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use discoverable plugins when you need to package reusable behavior outside the +Relay host binary. A package includes a `relay-plugin.toml` manifest and one of +two execution lanes: + +| Lane | Use when | Stable boundary | +| --- | --- | --- | +| `rust_dynamic` | Behavior must run in the Relay process. | Native ABI v1 | +| `worker` | Behavior should run in a separate local process. | `grpc-v1` | + +The manifest describes compatibility, capabilities, artifact integrity, and the +load contract. Operators register the manifest reference and component +configuration in `plugins.toml`, then enable the plugin through the CLI +lifecycle. Keep the package contract separate from the operator workflow: refer +operators to [Configure Discoverable +Plugins](/configure-plugins/discoverable-plugins). + +## Manifest Contract + +Every manifest uses the common fields shown below, followed by lane-specific +compatibility, capability, source, and load fields. This complete example +defines a Python worker: + +```toml +manifest_version = 1 + +[plugin] +id = "acme.example" +kind = "worker" + +[compat] +relay = ">=0.5,<1.0" +worker_protocol = "grpc-v1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_worker"] + +[source] +manifest_root = "." +artifact = "acme_example/worker.py" + +[integrity] +sha256 = "sha256:" + +[load] +runtime = "python" +entrypoint = "acme_example.worker:main" +``` + +`compat.relay` is a normal SemVer requirement. Use `>=0.5,<1.0` unless your +plugin requires a narrower Relay version. Keep `defaults.enabled = false`: +operators must enable a registered dynamic plugin explicitly. Declare only the +capabilities the plugin needs. Add `config_schema.path` only with the +`config_schema` capability. + +The following requirements vary by execution lane: + +| Manifest area | Native dynamic plugin | Worker plugin | +| --- | --- | --- | +| `plugin.kind` | `rust_dynamic` | `worker` | +| `compat` | `native_api = "1"` | `worker_protocol = "grpc-v1"` | +| `capabilities.items` | Includes `plugin_native` | Includes `plugin_worker` | +| `load` | `library` and `symbol` | `runtime` and `entrypoint` | +| `source.manifest_root` | Optional | Required for `runtime = "python"`; `nemo-relay plugins add` uses it to create and retain the managed worker environment. | + +Use `runtime = "python"` for a `module:function` entrypoint, `runtime = +"rust"` for a Rust executable, or `runtime = "command"` for another local +executable that implements `grpc-v1`. + +The CLI verifies `source.artifact` against `integrity.sha256` during `plugins +add` and `plugins validate`. After a native artifact or non-Python worker +changes, update its digest and rerun validation. After Python worker source or +dependency changes, remove and add the worker again so Relay rebuilds its +managed environment. Native loading also verifies `load.library` against +`integrity.sha256`. Add `integrity.signature` when an operator’s policy can +require Ed25519 signature verification. Treat these fields as release +artifacts: update the digest and signature whenever the declared artifact +changes. + +## Choose a Discoverable Plugin Type + +Discoverable plugins always use a manifest. Choose the guide that matches the +manifest-backed package you distribute: + +- **Native dynamic plugin (Rust):** Refer to [Native Dynamic Plugins (Rust)](/build-plugins/dynamic-plugins/native-dynamic/about) + for shared-library packages and the native ABI, then [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) + for the SDK example. +- **gRPC worker plugin:** Refer to [gRPC Worker Plugin Concepts](/build-plugins/dynamic-plugins/grpc-worker/about) + for runtime choices, lifecycle, and trust; [gRPC Worker Plugins (Rust)](/build-plugins/dynamic-plugins/grpc-worker/rust/about) + or [gRPC Worker Plugins (Python)](/build-plugins/dynamic-plugins/grpc-worker/python/about) + for language-specific authoring; and [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) + for the stable `grpc-v1` boundary. + +For the complete comparison of manifest-backed packages and in-process, +code-driven language binding plugins, refer to [Build Plugins](/build-plugins). diff --git a/docs/build-plugins/dynamic-plugins/grpc-worker/about.mdx b/docs/build-plugins/dynamic-plugins/grpc-worker/about.mdx new file mode 100644 index 000000000..abb068f39 --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/grpc-worker/about.mdx @@ -0,0 +1,88 @@ +--- +title: "gRPC Worker Plugin Concepts" +description: "Choose and package local gRPC worker plugins for NeMo Relay." +position: 12 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Worker dynamic plugins run outside the Relay process and install subscribers, +guardrails, and intercepts through the stable `grpc-v1` protocol. The host +starts the local worker, validates its component configuration, receives a +declarative registration plan, and installs proxy callbacks. + +## Choose a Worker Runtime + +Choose the runtime that matches the artifact you distribute: + +- Use `python` for a `module:function` entrypoint in a Relay-managed Python + environment. Include `source.manifest_root`, then run `nemo-relay plugins + add` so Relay can create and retain the required environment. +- Use `rust` for a manifest-relative or absolute Rust executable. +- Use `command` for another manifest-relative or absolute local executable that + implements `grpc-v1`. + +All worker runtimes receive the same local endpoint and activation credentials. +The process boundary isolates crashes and dependencies, but it does not create a +security sandbox. + +## Manifest + +Use the following worker manifest: + +```toml +manifest_version = 1 + +[plugin] +id = "acme.policy" +kind = "worker" + +[compat] +relay = ">=0.5,<1.0" +worker_protocol = "grpc-v1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_worker"] + +[source] +manifest_root = "." +artifact = "acme_policy/worker.py" + +[integrity] +sha256 = "sha256:" + +[load] +runtime = "python" +entrypoint = "acme_policy.worker:main" +``` + +Set `compat.worker_protocol` to `grpc-v1` and include `plugin_worker` in the +capability list. Refer to [Choose a Worker Runtime](#choose-a-worker-runtime) +for the runtime-specific `load` and `source` requirements. + +The worker receives its activation ID, plugin ID, worker and host endpoints, +and a local activation token through environment variables. Do not start the +worker directly during normal operation. The host supplies these values. + +## Registration and Trust + +Workers return declarative registrations. Relay owns registry mutation, +namespacing, rollback, and deregistration. Relay DTOs use `JsonEnvelope` +values, while protobuf handles control flow. + +Run `nemo-relay plugins add` and `nemo-relay plugins validate` to evaluate the +configured host policy and artifact trust evidence. Refer to [Configure +Discoverable Plugins](/configure-plugins/discoverable-plugins) for the +operator-side lifecycle and validation flow. + +## Next Steps + +- Refer to [gRPC Worker Plugins (Rust)](/build-plugins/dynamic-plugins/grpc-worker/rust/about) + or [gRPC Worker Plugins (Python)](/build-plugins/dynamic-plugins/grpc-worker/python/about) + for complete language-specific authoring workflows. +- Refer to [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) + when you implement `grpc-v1` without an SDK. diff --git a/docs/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol.mdx b/docs/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol.mdx new file mode 100644 index 000000000..872a6bde9 --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol.mdx @@ -0,0 +1,135 @@ +--- +title: "gRPC Worker Protocol Overview" +description: "Understand the local `grpc-v1` contract and canonical protocol definition for NeMo Relay worker plugins." +position: 15 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +`grpc-v1` is the stable out-of-process plugin protocol for +`plugin.kind = "worker"`. Python, Rust, and other local executables can +implement the `nemo.relay.worker.v1` API. Relay starts every worker and connects +to it through local endpoints. Remote worker endpoints are not supported. + +Use the Rust or Python SDK unless you need another runtime. Refer to [gRPC +Worker Plugin Concepts](/build-plugins/dynamic-plugins/grpc-worker/about) for runtime +choices and lifecycle guidance. + +## Service Contract + +Workers implement the `PluginWorker` service: + +- `Handshake` and `Health` identify a ready worker. +- `Validate` returns configuration diagnostics. +- `Register` returns declarative subscriber, guardrail, and intercept + registrations. +- `Invoke` and `InvokeStream` run registered behavior. +- `CancelInvocation` requests cancellation, and `Shutdown` requests process + termination. + +Relay implements `RelayHostRuntime` for worker-initiated operations. It lets a +worker emit marks, manage scopes and isolated scope stacks, and call tool, LLM, +or LLM-stream continuations during execution intercepts. + +This page summarizes the stable contract. Refer to the [canonical protocol +definition](https://github.com/NVIDIA/NeMo-Relay/blob/0.5.0-alpha.20260706/crates/worker-proto/proto/nemo/relay/worker/v1/plugin_worker.proto) +for every message and RPC field required to implement another runtime. + +## PluginWorker RPCs + +The worker implements the following RPCs: + +| RPC | Request and response | Purpose | +| --- | --- | --- | +| `Handshake` | `HandshakeRequest` → `HandshakeResponse` | Confirms plugin identity, `grpc-v1`, SDK/runtime details, and supported registration surfaces. | +| `Health` | `HealthRequest` → `HealthResponse` | Reports whether the worker is ready. | +| `Validate` | `ValidateRequest` → `ValidateResponse` | Validates the component `config` envelope and returns diagnostics or `WorkerError`. | +| `Register` | `RegisterRequest` → `RegisterResponse` | Returns declarative registrations or `WorkerError`. | +| `Invoke` | `InvokeRequest` → `InvokeResponse` | Runs a subscriber, guardrail, or non-streaming intercept. | +| `InvokeStream` | `InvokeRequest` → `stream StreamChunk` | Runs a streaming LLM intercept. | +| `CancelInvocation` | `CancelInvocationRequest` → `WorkerAck` | Requests cooperative cancellation of an invocation. | +| `Shutdown` | `ShutdownRequest` → `WorkerAck` | Requests worker shutdown after Relay removes its proxy callbacks. | + +Every request carries the activation ID and authentication token. `Validate` +and `Register` also carry the plugin ID and component configuration. + +## Registration and Invocation + +On success, `RegisterResponse` contains `Registration` records with a local +name, surface, priority, and `break_chain` value. A failed registration can +return `WorkerError` without registrations. The supported surfaces are: + +- `SUBSCRIBER` +- `TOOL_SANITIZE_REQUEST_GUARDRAIL`, `TOOL_SANITIZE_RESPONSE_GUARDRAIL`, + `TOOL_CONDITIONAL_EXECUTION_GUARDRAIL`, `TOOL_REQUEST_INTERCEPT`, and + `TOOL_EXECUTION_INTERCEPT` +- `LLM_SANITIZE_REQUEST_GUARDRAIL`, `LLM_SANITIZE_RESPONSE_GUARDRAIL`, + `LLM_CONDITIONAL_EXECUTION_GUARDRAIL`, `LLM_REQUEST_INTERCEPT`, + `LLM_EXECUTION_INTERCEPT`, and `LLM_STREAM_EXECUTION_INTERCEPT` + +`InvokeRequest` identifies the registration, surface, invocation, optional +continuation, and scope context. Its payload is one of an event, tool +invocation, or LLM invocation. `InvokeResponse` returns an empty result, JSON +result, guardrail result, LLM request-intercept result, tool-execution result, +or `WorkerError`. `InvokeStream` emits JSON chunks or `WorkerError` chunks. + +## RelayHostRuntime RPCs + +Relay implements these RPCs for worker callbacks: + +| RPC group | RPCs | +| --- | --- | +| Marks and scopes | `EmitMark`, `PushScope`, `PopScope` | +| Isolated scope stacks | `CreateScopeStack`, `DropScopeStack` | +| Execution continuations | `ToolNext`, `LlmNext`, `LlmStreamNext` | + +Every host-runtime request also carries the activation ID and authentication +token. Scope operations include a `ScopeContext`; continuation calls include +the continuation ID that Relay supplied for the active intercept. + +## Authentication and Endpoints + +Relay supplies an activation ID, an activation token, the worker endpoint, and +the host endpoint when it starts the process. SDKs attach the activation ID and +token to protocol calls. The host rejects requests with an invalid activation ID +or token. + +On Unix platforms, Relay uses local Unix sockets. On other platforms, Relay +uses loopback TCP endpoints. Workers must not accept arbitrary remote endpoint +configuration. + +## Payloads + +Relay data values use this envelope: + +```proto +message JsonEnvelope { + string schema = 1; + bytes json = 2; +} +``` + +Use `JsonEnvelope` for Relay DTOs. Protobuf defines protocol control flow and +does not duplicate Relay event, tool, LLM, scope, or diagnostic models. + +## Activation and Shutdown + +1. Run `nemo-relay plugins validate ` before enabling or running a + plugin to check its manifest, trust evidence, and optional static + configuration schema. +2. Relay creates local endpoints, starts the worker, and completes health and + handshake checks. +3. Relay sends component configuration to `Validate`. Error diagnostics stop + initialization after the worker process starts. +4. Relay obtains registrations from `Register` and installs proxy callbacks. + Relay rolls back installed callbacks if initialization fails. +5. Relay removes proxy callbacks before it sends `Shutdown` to the worker. + +## Errors + +gRPC status communicates transport, authentication, malformed protocol +requests, and some stream failures. Registration and unary callback failures +can return structured `WorkerError` values. A stream callback failure can +terminate the gRPC stream with a status error. Worker implementations must +handle both error forms. diff --git a/docs/build-plugins/dynamic-plugins/grpc-worker/python/about.mdx b/docs/build-plugins/dynamic-plugins/grpc-worker/python/about.mdx new file mode 100644 index 000000000..6317330a5 --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/grpc-worker/python/about.mdx @@ -0,0 +1,304 @@ +--- +title: "gRPC Worker Plugins (Python)" +description: "Build, package, configure, and run Python gRPC worker plugins for NeMo Relay." +position: 14 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the `nemo-relay-plugin` package to build an out-of-process plugin worker. +The SDK owns gRPC server setup, generated protocol stubs, JSON-envelope +conversion, continuations, cancellation hooks, and scope-stack helpers. +Refer to [gRPC Worker Plugin Concepts](/build-plugins/dynamic-plugins/grpc-worker/about) +to compare the Python, Rust, and command worker runtimes. + +This guide builds a complete plugin that adds a tag to every tool request and +emits a mark through the Relay host runtime. + +## Create the Project + +Create the following project layout: + +```text +python-grpc-worker/ +├── pyproject.toml +├── relay-plugin.toml +└── nemo_relay_python_grpc_worker/ + ├── __init__.py + └── worker.py +``` + +Add the following `pyproject.toml` file: + +```toml +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[project] +name = "nemo-relay-python-grpc-worker-example" +version = "0.1.0" +description = "Example Python gRPC worker plugin for NeMo Relay" +requires-python = ">=3.11" +dependencies = [ + "nemo-relay-plugin>=0.5.0", +] + +[tool.setuptools.packages.find] +where = ["."] +include = ["nemo_relay_python_grpc_worker"] +``` + +Create `nemo_relay_python_grpc_worker/__init__.py` with the following content: + +```python +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Example Python gRPC worker plugin package.""" +``` + +## Implement the Worker + +Add the following implementation to +`nemo_relay_python_grpc_worker/worker.py`: + +```python +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Example Python worker plugin using the nemo-relay-plugin SDK.""" + +from __future__ import annotations + +from typing import Any + +from nemo_relay_plugin import ConfigDiagnostic, DiagnosticLevel, Json, PluginContext, WorkerPlugin, serve_plugin + + +class ExamplePythonWorker(WorkerPlugin): + """Small worker plugin that tags tool request JSON and emits a host mark.""" + + plugin_id = "examples.python_grpc_worker" + + def validate(self, config: Json) -> list[ConfigDiagnostic | dict[str, Any]]: + if not isinstance(config, dict): + return [ + ConfigDiagnostic( + level=DiagnosticLevel.ERROR, + code="examples.python_grpc_worker.invalid_config", + component=self.plugin_id, + message="plugin config must be a JSON object", + ) + ] + if config.get("reject") is True: + return [ + ConfigDiagnostic( + level=DiagnosticLevel.ERROR, + code="examples.python_grpc_worker.rejected", + component=self.plugin_id, + field="reject", + message="Python gRPC worker rejection requested", + ) + ] + if "tag" in config and not isinstance(config["tag"], str): + return [ + ConfigDiagnostic( + level=DiagnosticLevel.ERROR, + code="examples.python_grpc_worker.invalid_tag", + component=self.plugin_id, + field="tag", + message="tag must be a string", + ) + ] + return [] + + def register(self, ctx: PluginContext, config: Json) -> None: + if not isinstance(config, dict): + raise TypeError("plugin config must be a JSON object") + if config.get("reject") is True: + raise ValueError("Python gRPC worker rejection requested") + tag = config.get("tag", "python_grpc_worker") + if not isinstance(tag, str): + raise TypeError("tag must be a string") + + async def tag_tool_request(tool_name: str, args: Json) -> Json: + tagged_args = _tag_json(args, tag) + await ctx.runtime.emit_mark( + "examples.python_grpc_worker.tool_request", + {"tool_name": tool_name, "source": "python-grpc-worker", "tag": tag}, + ) + return tagged_args + + ctx.register_tool_request_intercept("tag_tool_request", tag_tool_request) + + +def _tag_json(value: Json, tag: str) -> Json: + if not isinstance(value, dict): + return value + metadata = value.get("_nemo_relay_plugin") + if metadata is None: + metadata = {} + elif not isinstance(metadata, dict): + return value + return { + **value, + "_nemo_relay_plugin": {**metadata, "tag": tag}, + } + + +async def main() -> None: + """Entrypoint referenced by relay-plugin.toml.""" + await serve_plugin(ExamplePythonWorker()) + + +if __name__ == "__main__": + import asyncio + + asyncio.run(main()) +``` + +`WorkerPlugin.validate` returns structured diagnostics after Relay starts the +worker process and before Relay installs registrations. `WorkerPlugin.register` +registers a tool request intercept that updates the request JSON and emits a +mark through `PluginContext.runtime`. The `main` entrypoint blocks until Relay +requests shutdown. + +## Create the Manifest + +Create `relay-plugin.toml` with the following content: + +```toml +manifest_version = 1 + +[plugin] +id = "examples.python_grpc_worker" +kind = "worker" + +[compat] +relay = ">=0.5,<1.0" +worker_protocol = "grpc-v1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_worker"] + +[source] +manifest_root = "." +artifact = "nemo_relay_python_grpc_worker/worker.py" + +[integrity] +sha256 = "sha256:966849be254cc6299a17a4bb65500363e9a48f98cc1e0091192e42b23821486f" + +[load] +runtime = "python" +entrypoint = "nemo_relay_python_grpc_worker.worker:main" +``` + +The digest matches the `worker.py` file in this guide. Use Python 3.11 or later +to calculate a new digest before you register the worker: + +```bash +python3 - <<'PY' +from hashlib import sha256 +from pathlib import Path + +artifact = Path("nemo_relay_python_grpc_worker/worker.py") +print(f"sha256:{sha256(artifact.read_bytes()).hexdigest()}") +PY +``` + +## Register the Plugin + +Run the following commands from the project directory: + +```bash +relay_tmp="$(mktemp -d)" +relay_config="$relay_tmp/gateway.toml" + +nemo-relay --config "$relay_config" plugins add ./relay-plugin.toml +``` + +`plugins add` creates an isolated Relay-managed Python environment and installs +the `source.manifest_root` project into it. Relay uses that recorded environment +when it starts the worker. Do not start the worker directly: Relay supplies its +worker socket, host socket, activation ID, and activation token. + +## Configure the Plugin + +After `plugins add` registers the worker, edit +`$relay_tmp/plugins.toml`. Add the configuration table to the +`[[plugins.dynamic]]` entry that `plugins add` created. The configuration sets +the tag that the worker adds to tool request JSON: + +```toml +# `plugins add` writes the canonical absolute manifest path. +[[plugins.dynamic]] +manifest = "/absolute/path/to/python-grpc-worker/relay-plugin.toml" + +[plugins.dynamic.config] +tag = "documentation" +``` + +Do not use this TOML block instead of `plugins add` for a Python worker. The +CLI also creates the lifecycle record that points to the Relay-managed Python +environment. The `tag` field is optional and defaults to `python_grpc_worker`. + +## Test Worker-Side Validation + +`nemo-relay plugins validate` checks the manifest, trust evidence, and optional +static schema. It does not call `WorkerPlugin.validate`. + +To test runtime validation: + +1. Set `reject = true`, then start the gateway. Relay starts the worker and + reports the worker diagnostic before it installs registrations. +2. Set `tag` to a non-string value, then start the gateway. Relay reports the + structured diagnostic from `WorkerPlugin.validate`. + +`WorkerPlugin.validate` runs after Relay starts the worker and before Relay +installs registrations. + +## Run the Plugin + +Enable, validate, and start the gateway with the following commands: + +```bash +nemo-relay --config "$relay_config" plugins enable examples.python_grpc_worker +nemo-relay --config "$relay_config" plugins validate examples.python_grpc_worker +nemo-relay --config "$relay_config" --bind 127.0.0.1:4040 +``` + +## Update the Worker + +After you change worker source code or dependencies, rebuild the managed +environment instead of only updating the manifest digest: + +1. Recalculate `integrity.sha256` with Python 3.11 or later and update + `relay-plugin.toml`. +2. Remove the registered worker, then add it again: + + ```bash + nemo-relay --config "$relay_config" plugins remove examples.python_grpc_worker + nemo-relay --config "$relay_config" plugins add ./relay-plugin.toml + ``` + +3. Restore any `[[plugins.dynamic]].config` values, then enable the worker. + +`plugins add` installs a new copy of `source.manifest_root` into a fresh +Relay-managed environment. It cannot refresh an already registered worker. + +After you stop Relay, remove the plugin and its managed environment: + +```bash +nemo-relay --config "$relay_config" plugins remove examples.python_grpc_worker +rm -rf "$relay_tmp" +``` + +Refer to [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) for the +shared `grpc-v1` contract and +[Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) for +host trust and lifecycle policies. diff --git a/docs/build-plugins/dynamic-plugins/grpc-worker/rust/about.mdx b/docs/build-plugins/dynamic-plugins/grpc-worker/rust/about.mdx new file mode 100644 index 000000000..9c5fa114f --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/grpc-worker/rust/about.mdx @@ -0,0 +1,141 @@ +--- +title: "gRPC Worker Plugins (Rust)" +description: "Build, package, and register out-of-process Rust gRPC worker plugins for NeMo Relay." +position: 13 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the `nemo-relay-worker` crate to build an out-of-process Rust plugin. The +SDK implements the `grpc-v1` service, exchanges JSON envelopes with Relay, +handles cancellation, and provides typed registration and host-runtime helpers. +Refer to [gRPC Worker Plugin Concepts](/build-plugins/dynamic-plugins/grpc-worker/about) +to compare the Rust, Python, and command worker runtimes. + +## Create the Project + +Create a binary Rust project with the following `Cargo.toml` file: + +```toml +[package] +name = "examples-rust-worker" +version = "0.1.0" +edition = "2024" + +[dependencies] +nemo-relay-worker = "0.5.0" +tokio = { version = "1", features = ["macros", "rt-multi-thread"] } +``` + +## Implement the Worker + +Add the following implementation to `src/main.rs`: + +```rust +use nemo_relay_worker::{Json, PluginContext, Result, WorkerPlugin, serve_plugin}; + +struct ExampleWorker; + +impl WorkerPlugin for ExampleWorker { + fn plugin_id(&self) -> &str { + "examples.rust_worker" + } + + fn validate(&self, config: &Json) -> Vec { + let _ = config; + Vec::new() + } + + fn register(&self, context: &mut PluginContext, _config: &Json) -> Result<()> { + context.register_subscriber("example", |event| { + let _ = event.name(); + }); + Ok(()) + } +} + +#[tokio::main] +async fn main() -> Result<()> { + serve_plugin(ExampleWorker).await +} +``` + +Relay provides the worker and host endpoints, activation ID, plugin ID, and +activation token through environment variables. The Rust SDK derives plugin +identity from `WorkerPlugin::plugin_id()`; custom launchers can use +`NEMO_RELAY_PLUGIN_ID`. `serve_plugin` consumes the endpoints and activation +credentials and runs until Relay requests shutdown. Do not start the worker +directly for normal operation. + +## Package the Worker + +Create `relay-plugin.toml` with the following content. The artifact digest must +match the executable that operators receive: + +```toml +manifest_version = 1 + +[plugin] +id = "examples.rust_worker" +kind = "worker" + +[compat] +relay = ">=0.5,<1.0" +worker_protocol = "grpc-v1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_worker"] + +[source] +artifact = "target/release/examples-rust-worker" + +[integrity] +sha256 = "sha256:" + +[load] +runtime = "rust" +entrypoint = "target/release/examples-rust-worker" +``` + +Build the executable with the following command: + +```bash +cargo build --release +``` + +Calculate the SHA-256 digest of the executable with the following command: +This command requires Python 3. + + +Use the following PowerShell command on Windows: + +```powershell +$artifact = ".\\target\\release\\examples-rust-worker.exe" +$hash = (Get-FileHash -Algorithm SHA256 $artifact).Hash.ToLower() +"sha256:$hash" +``` + +Replace `sha256:` with the output before you register the +plugin. On Windows, use the `.exe` artifact name in `source.artifact`, +`load.entrypoint`, and the digest command. + +## Register the Worker + +Register, enable, and validate the worker with the following commands: + +```bash +nemo-relay plugins add ./relay-plugin.toml +nemo-relay plugins enable examples.rust_worker +nemo-relay plugins validate examples.rust_worker +``` + +## Related Topics + +- Refer to [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) + for lifecycle and trust-policy configuration. +- Refer to [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) + for the lower-level protocol contract. diff --git a/docs/build-plugins/dynamic-plugins/native-dynamic/about.mdx b/docs/build-plugins/dynamic-plugins/native-dynamic/about.mdx new file mode 100644 index 000000000..602fcc3b6 --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/native-dynamic/about.mdx @@ -0,0 +1,157 @@ +--- +title: "Native Dynamic Plugins (Rust)" +description: "Build in-process Rust shared-library plugins against the NeMo Relay Native ABI v1." +position: 10 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Native dynamic plugins are trusted in-process shared libraries. Use them when +plugin behavior needs host-process access while still following the Relay plugin +contract: a stable kind, JSON component configuration, validation diagnostics, +and registration through a component-scoped context. + + +Native plugins are not sandboxed. They run in the gateway process, must not +unwind across ABI callbacks, and remain loaded until Relay removes their +registered callbacks. + + +## Manifest + +Use the following manifest: + +```toml +manifest_version = 1 + +[plugin] +id = "acme.native_policy" +kind = "rust_dynamic" + +[compat] +relay = ">=0.5,<1.0" +native_api = "1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_native"] + +[source] +artifact = "target/release/libacme_native_policy.dylib" + +[integrity] +sha256 = "sha256:" + +[load] +library = "target/release/libacme_native_policy.dylib" +symbol = "nemo_relay_register_plugin" +``` + +Set `plugin.kind` to `rust_dynamic`, `compat.native_api` to `"1"`, and +`capabilities.items` to include `plugin_native`. Relay resolves relative paths +from the manifest. The exported symbol must return a descriptor whose +`plugin_kind` matches `plugin.id` exactly. + +## Create a Native Plugin + +Create a Rust library project with the following `Cargo.toml` file: + +```toml +[package] +name = "acme-native-policy" +version = "0.1.0" +edition = "2024" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +nemo-relay-plugin = "0.5.0" +serde_json = "1" +``` + +Add the following implementation to `src/lib.rs`: + +```rust +use nemo_relay_plugin::{Json, NativePlugin, PluginContext, Result}; +use serde_json::Map; + +struct NativePolicy; + +impl NativePlugin for NativePolicy { + fn plugin_kind(&self) -> &str { + "acme.native_policy" + } + + fn register( + &mut self, + _config: &Map, + context: &mut PluginContext<'_>, + ) -> Result<()> { + context.register_subscriber("audit", |_| {})?; + Ok(()) + } +} + +nemo_relay_plugin::nemo_relay_plugin!(nemo_relay_register_plugin, || NativePolicy); +``` + +Build the library with the following command: + +```bash +cargo build --release +``` + +Update `source.artifact` and `load.library` with the resulting platform library +path, then replace `` with that library's SHA-256 digest. Use +`.dylib` on macOS, `.so` on Linux, or `.dll` on Windows. Refer to [Build a Rust +Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) for a complete example +with validation, middleware, scopes, and configuration schema support. + +## Native ABI v1 + +The host passes a `NemoRelayNativeHostApiV1` table to the entry symbol. The +plugin returns a `NemoRelayNativePluginV1` descriptor: + +```rust +extern "C" fn nemo_relay_register_plugin( + host: *const NemoRelayNativeHostApiV1, + out: *mut NemoRelayNativePluginV1, +) -> NemoRelayStatus +``` + +Text and JSON data cross this boundary as host-owned +`NemoRelayNativeString` handles. ABI structs also carry scalars, opaque +handles, callback pointers, and plugin-owned `user_data`. Do not pass Rust +runtime types, trait objects, futures, `serde_json::Value`, or allocator-owned +strings across the ABI. + +ABI callbacks can register these runtime surfaces: + +- Subscribers +- Tool and LLM guardrails or intercepts, including stream intercepts +- Marks, scopes, and isolated scope stacks + +Relay keeps the library alive while those registrations exist and deregisters +them before unloading it. + +Use the `nemo-relay-plugin` crate rather than the host `nemo-relay` runtime +crate. Refer to [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) +for the SDK-backed example. + +## Register the Plugin + +After you package the manifest and library, register and validate the plugin: + +```bash +nemo-relay plugins add ./relay-plugin.toml +nemo-relay plugins enable acme.native_policy +nemo-relay plugins validate acme.native_policy +``` + +Refer to [Configure Discoverable +Plugins](/configure-plugins/discoverable-plugins) for lifecycle and trust-policy +configuration. diff --git a/docs/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example.mdx b/docs/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example.mdx new file mode 100644 index 000000000..47495d106 --- /dev/null +++ b/docs/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example.mdx @@ -0,0 +1,133 @@ +--- +title: "Build a Rust Native Plugin" +description: "Build and package the NeMo Relay Rust native dynamic-plugin example." +position: 11 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the `nemo-relay-plugin` crate to create an in-process native plugin without +writing the C ABI by hand. The SDK exports the stable ABI entry point and +provides typed helpers for the supported registration surfaces. + +This example builds a native plugin that registers an event subscriber. The +repository also includes an extended example in `examples/rust-native-plugin` +with validation, middleware, marks, scopes, and a configuration schema. + +## Create the Project + +Create a Rust library project with this `Cargo.toml` file: + +```toml +[package] +name = "acme-native-policy" +version = "0.1.0" +edition = "2024" + +[lib] +crate-type = ["cdylib"] + +[dependencies] +nemo-relay-plugin = "0.5.0" +serde_json = "1" +``` + +Add the following implementation to `src/lib.rs`: + +```rust +use nemo_relay_plugin::{Json, NativePlugin, PluginContext, Result}; +use serde_json::Map; + +struct NativePolicy; + +impl NativePlugin for NativePolicy { + fn plugin_kind(&self) -> &str { + "acme.native_policy" + } + + fn register( + &mut self, + _config: &Map, + context: &mut PluginContext<'_>, + ) -> Result<()> { + context.register_subscriber("audit", |_| {}) + } +} + +nemo_relay_plugin::nemo_relay_plugin!(nemo_relay_register_plugin, || NativePolicy); +``` + +## Create the Manifest + +Create `relay-plugin.toml` with the following content: + +```toml +manifest_version = 1 + +[plugin] +id = "acme.native_policy" +kind = "rust_dynamic" + +[compat] +relay = ">=0.5,<1.0" +native_api = "1" + +[defaults] +enabled = false + +[capabilities] +items = ["plugin_native"] + +[source] +artifact = "target/release/" + +[integrity] +sha256 = "sha256:" + +[load] +library = "target/release/" +symbol = "nemo_relay_register_plugin" +``` + +Update the library filename for your platform: use `.dylib` on macOS, `.so` +on Linux, or `.dll` on Windows. Replace `` in both +manifest fields with that filename. + +## Build, Register, and Validate + +Build the library, calculate its digest, and register the plugin with the +following commands. The digest command requires Python 3. + +```bash +cargo build --release +python - <<'PY' +from hashlib import sha256 +from pathlib import Path + +artifact = Path("target/release/") +print(f"sha256:{sha256(artifact.read_bytes()).hexdigest()}") +PY +nemo-relay plugins add ./relay-plugin.toml +nemo-relay plugins enable acme.native_policy +nemo-relay plugins validate acme.native_policy +``` + +Use the following PowerShell command on Windows: + +```powershell +$artifact = ".\\target\\release\\" +$hash = (Get-FileHash -Algorithm SHA256 $artifact).Hash.ToLower() +"sha256:$hash" +``` + +Replace `` in the digest command, then replace +`sha256:` with its output before you run `plugins add`. The +command writes a `[[plugins.dynamic]]` manifest reference. Use +`nemo-relay plugins edit` or edit that record to add component fields when the +plugin accepts configuration. + +Keep tests outside `src`, use the shared DTOs re-exported by +`nemo-relay-plugin`, and do not retain host-owned ABI handles after a callback +returns. Refer to [Configure Discoverable +Plugins](/configure-plugins/discoverable-plugins) for trust-policy configuration. diff --git a/docs/build-plugins/basic-guide.mdx b/docs/build-plugins/language-binding/about.mdx similarity index 61% rename from docs/build-plugins/basic-guide.mdx rename to docs/build-plugins/language-binding/about.mdx index 944f3c4c5..6e8e1a306 100644 --- a/docs/build-plugins/basic-guide.mdx +++ b/docs/build-plugins/language-binding/about.mdx @@ -1,6 +1,6 @@ --- -title: "Define a Plugin" -description: "" +title: "Language Binding Plugins" +description: "Build in-process, code-driven NeMo Relay plugins in Rust, Python, or Node.js." position: 2 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -9,11 +9,21 @@ import { MermaidStyles } from "@/components/MermaidStyles"; SPDX-License-Identifier: Apache-2.0 */} -Use this guide when you want to package reusable NeMo Relay behavior as a plugin that can be activated from configuration. +Use this guide to build an in-process, code-driven plugin in Rust, Python, or +Node.js. The application registers the plugin kind with the loaded language +binding, then initializes it with the same component configuration model used +by built-in plugins. + +This model does not load a shared library or start a worker process. For those +models, use [Native Dynamic Plugins (Rust)](/build-plugins/dynamic-plugins/native-dynamic/about), +[gRPC Worker Plugins (Rust)](/build-plugins/dynamic-plugins/grpc-worker/rust/about), or [gRPC +Worker Plugins (Python)](/build-plugins/dynamic-plugins/grpc-worker/python/about). ## What You Build -You will define the plugin's purpose, stable kind name, configuration boundary, runtime surfaces, and activation lifecycle. The result is a small plugin contract that can be validated and registered through the more focused follow-on guides. +Define the plugin's purpose, stable kind name, configuration boundary, runtime +surfaces, and activation lifecycle. Then use the focused guides to validate and +register the resulting plugin contract. NeMo Relay plugin configuration keys use `snake_case` in every language and file @@ -42,27 +52,24 @@ A plugin needs a stable shape before operators can activate it from config: | Stable `kind` | The plugin registry uses this string to match config to implementation. | | JSON-compatible config | Config must move across Python, Node.js, Rust, files, tests, and deployment systems. | | Validation hook | Operators need diagnostics before runtime behavior changes. | -| Registration hook | Runtime behavior should be installed through `PluginContext` for name qualification and rollback. | -| Runtime ownership | The plugin should clearly own subscribers, middleware, adaptive behavior, or a small bundle of related surfaces. | +| Registration hook | Register runtime behavior through `PluginContext` so Relay can qualify names and roll back failed setup. | +| Runtime ownership | The plugin should clearly own subscribers, middleware, or a small bundle of related surfaces. | -Keep runtime objects out of config. Provider clients, callbacks, file handles, caches, and credentials should be created inside plugin code or resolved from safe references during registration. +Keep runtime objects out of config. Create provider clients, callbacks, file +handles, caches, and credentials in plugin code, or resolve them from safe +references during registration. ## What a Plugin Can Install A plugin can install one or more of these runtime surfaces: -- Subscribers -- Tool sanitize-request guardrails -- Tool sanitize-response guardrails -- Tool conditional-execution guardrails -- Tool request intercepts -- Tool execution intercepts -- LLM sanitize-request guardrails -- LLM sanitize-response guardrails -- LLM conditional-execution guardrails -- LLM request intercepts -- LLM execution intercepts -- LLM stream execution intercepts +- **Subscribers:** Event subscribers. +- **Tool middleware:** Sanitize-request and sanitize-response guardrails, + conditional-execution guardrails, request intercepts, and execution + intercepts. +- **LLM middleware:** Sanitize-request and sanitize-response guardrails, + conditional-execution guardrails, request intercepts, execution intercepts, + and stream execution intercepts. Start with one surface. Add a bundle only when one configuration document clearly controls related behavior, such as a subscriber plus the request intercepts needed to add correlation metadata. @@ -93,7 +100,10 @@ flowchart TB Context -->|registration error| Rollback ``` -The lifecycle is staged: register the plugin kind, validate component config, initialize enabled components, and let `PluginContext` install runtime behavior. If registration fails partway through, the plugin system can roll back partial setup. +The lifecycle works in stages. Register the plugin kind, validate component +config, and initialize enabled components. `PluginContext` installs runtime +behavior. If registration fails partway through, Relay rolls back the partial +setup. ## Keep the First Plugin Small @@ -104,11 +114,16 @@ The easiest first plugin is one of these: - A sanitize guardrail plugin that redacts one field family. - A policy plugin that registers one conditional-execution guardrail. -Avoid a first plugin that combines unrelated subscribers, request transforms, policy checks, and adaptive behavior. Multi-surface bundles are useful later, but they need stronger validation and rollout controls. +Avoid a first plugin that combines unrelated subscribers, request transforms, and +policy checks. Multi-surface bundles are useful later, but they need stronger +validation and rollout controls. Refer to [Adaptive Configuration](/configure-plugins/adaptive/configuration) +when you need Adaptive behavior. ## Minimal Config Contract -The top-level config document has `version`, `components`, and `policy`. Each component chooses a plugin kind and passes component-local JSON config to that plugin. +The top-level config document has `version`, `components`, and `policy`. Each +component chooses a plugin kind and passes component-local JSON configuration to +that plugin. The following document shows the complete shape: ```json { @@ -149,7 +164,7 @@ Before you write the plugin implementation, answer these questions: Use these links to continue from this workflow into the next related task. -- Define validation behavior with [Validate Plugin Configuration](/build-plugins/validate-configuration). -- Register runtime behavior with [Register Plugin Behavior](/build-plugins/register-behavior). -- Add rollout controls with [Design Plugin Configuration](/build-plugins/advanced-configuration). -- Review complete examples in [Code Examples](/build-plugins/code-examples). +- Define validation behavior with [Validate Plugin Configuration](/build-plugins/language-binding/validate-configuration). +- Register runtime behavior with [Register Plugin Behavior](/build-plugins/language-binding/register-behavior). +- Add rollout controls with [Design Plugin Configuration](/build-plugins/language-binding/advanced-configuration). +- Review complete examples in [Code Examples](/build-plugins/language-binding/code-examples). diff --git a/docs/build-plugins/advanced-configuration.mdx b/docs/build-plugins/language-binding/advanced-configuration.mdx similarity index 71% rename from docs/build-plugins/advanced-configuration.mdx rename to docs/build-plugins/language-binding/advanced-configuration.mdx index 66fa13107..a1b75d68d 100644 --- a/docs/build-plugins/advanced-configuration.mdx +++ b/docs/build-plugins/language-binding/advanced-configuration.mdx @@ -1,16 +1,20 @@ --- title: "Design Plugin Configuration" -description: "" +description: "Design stable configuration, validation, rollout, and lifecycle behavior for NeMo Relay plugins." position: 6 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -Use this guide when a plugin needs more than a single flag or string to configure it safely. +Use this guide to safely configure a plugin that needs more than a single flag +or string. ## What You Design -You will define the plugin's configuration contract, validation rules, advanced configuration patterns, and runtime registration plan. The goal is to make plugin activation predictable for operators while keeping runtime objects and business logic inside the plugin implementation. +Define the plugin's configuration contract, validation rules, advanced +configuration patterns, and runtime registration plan. Keep activation +predictable for operators while keeping runtime objects and business logic in +the plugin implementation. ## Plugin Shape and Requirements @@ -52,18 +56,25 @@ Keep the component config portable: - Put clients, callbacks, file handles, and provider SDK objects in plugin code, not config. - Include a component-local config version when the plugin's own schema needs to evolve independently. - Prefer references to secrets or endpoints over embedding sensitive values directly. -- Treat `enabled: false` as disabled for activation, not as a reason to skip validation. +- Treat a component whose `enabled` field has the value `false` as disabled for + activation, not as a reason to skip validation. + +The top-level `policy` controls validation that Relay handles. For a custom +plugin's component-local `config`, define unknown-field and unsupported-value +behavior in the plugin's `validate()` method. ## Configuration Validation -Validation should be deterministic and side-effect free. It should inspect config and return diagnostics; it should not register middleware, open network connections, create clients, or mutate process state. +Keep validation deterministic and side-effect free. Inspect configuration and +return diagnostics. Do not register middleware, open network connections, +create clients, or change process state. Check these areas: - Required fields are present. - Field types match the supported shape. -- Unknown fields are reported according to policy. -- Known fields have supported values. +- The plugin reports unknown component-local fields with stable diagnostics. +- The plugin reports unsupported component-local values with stable diagnostics. - Cross-field combinations make sense. - Environment-specific limitations are warnings unless they would make activation fail. @@ -93,7 +104,8 @@ Use a field such as `config.version` when the plugin's config schema needs indep ### Multiple Component Instances -When a plugin can be instantiated more than once, require explicit instance identity in config: +If your plugin supports multiple instances, require an explicit instance +identity in configuration: ```json { @@ -106,7 +118,9 @@ When a plugin can be instantiated more than once, require explicit instance iden } ``` -Use the instance identity in logs, diagnostics, and downstream resource names. Let the NeMo Relay plugin system qualify runtime registration names; do not hand-build global names to avoid collisions. +Use the instance identity in logs, diagnostics, and downstream resource names. +Let the NeMo Relay plugin system qualify runtime registration names. Do not +hand-build global names to avoid collisions. ### Presets and Overrides @@ -149,7 +163,9 @@ Use `PluginContext` to register: - LLM guardrails - LLM request, execution, and stream execution intercepts -The context gives the plugin system enough information to qualify runtime names and roll back partial setup if registration fails. Put all runtime registration work inside the registration hook so rollback can clean up correctly. +The context gives the plugin system enough information to qualify runtime names +and roll back partial setup if registration fails. Put all runtime registration +work in the registration hook so rollback can clean up correctly. Avoid these patterns: @@ -165,7 +181,7 @@ Before publishing a plugin config contract: 1. Validate the smallest correct config. 2. Validate a config with each required field missing. 3. Validate each unsupported enum or mode. -4. Validate unknown fields under each supported policy. +4. Validate unknown component-local fields and their diagnostic levels. 5. Initialize a valid config and confirm expected middleware or subscribers are active. 6. Force a registration failure and confirm partial setup is rolled back. @@ -173,7 +189,7 @@ Before publishing a plugin config contract: Use these links to continue from this workflow into the next related task. -- Build the first plugin with [Define a Plugin](/build-plugins/basic-guide). -- Validate plugin config with [Validate Plugin Configuration](/build-plugins/validate-configuration). -- Register runtime behavior with [Register Plugin Behavior](/build-plugins/register-behavior). -- Review reusable patterns in [Code Examples](/build-plugins/code-examples). +- Build the first plugin with [Language Binding Plugins](/build-plugins/language-binding/about). +- Validate plugin config with [Validate Plugin Configuration](/build-plugins/language-binding/validate-configuration). +- Register runtime behavior with [Register Plugin Behavior](/build-plugins/language-binding/register-behavior). +- Review reusable patterns in [Code Examples](/build-plugins/language-binding/code-examples). diff --git a/docs/build-plugins/language-binding/code-examples.mdx b/docs/build-plugins/language-binding/code-examples.mdx new file mode 100644 index 000000000..84346dc69 --- /dev/null +++ b/docs/build-plugins/language-binding/code-examples.mdx @@ -0,0 +1,363 @@ +--- +title: "Code Examples" +description: "Explore reusable NeMo Relay plugin patterns for request interception, event logging, and policy bundles." +position: 8 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +This page shows reusable Python and Node.js plugin patterns for request +interception, event logging, multi-surface policies, and framework-neutral +behavior. The same plugin contract applies to Rust. Refer to the [Header Plugin +Example](/build-plugins/language-binding/register-behavior#header-plugin-example) +for Rust-specific registration guidance. + +## Dynamic Header Injection + +Use an LLM request intercept when a plugin needs to inject tenant or routing metadata into every provider request. + +LLM request intercepts receive `name`, `request`, and `annotated`. Python +intercepts receive an immutable request, so they return a new request with the +required changes. Node.js intercepts receive a normal JavaScript request +object; return a complete outcome containing the intended request rather than +relying on in-place mutation. Rust intercepts receive a mutable request. + +The following complete examples define, register, validate, and activate a +plugin that adds a request header. Each example clears the active configuration +during shutdown. + + + +```python +import asyncio +from typing import Any + +import nemo_relay + +class HeaderPlugin: + def validate(self, plugin_config: dict[str, Any]) -> list[dict[str, str]]: + diagnostics = [] + for field in ("header_name", "value"): + if not isinstance(plugin_config.get(field), str): + diagnostics.append({ + "level": "error", + "code": "header-plugin.invalid_config", + "component": "header-plugin", + "field": field, + "message": f"{field} must be a string", + }) + return diagnostics + + def register(self, plugin_config: dict[str, Any], context: nemo_relay.plugin.PluginContext): + def add_header( + name: str, + request: nemo_relay.LLMRequest, + annotated: nemo_relay.AnnotatedLLMRequest | None + ) -> nemo_relay.LLMRequestInterceptOutcome: + # Return a new request with updated headers. + headers = request.headers.copy() + headers[plugin_config["header_name"]] = plugin_config["value"] + return nemo_relay.LLMRequestInterceptOutcome( + nemo_relay.LLMRequest(headers=headers, content=request.content), + annotated, + ) + + context.register_llm_request_intercept("inject-header", 100, False, add_header) + +async def main() -> None: + nemo_relay.plugin.register("header-plugin", HeaderPlugin()) + + config = nemo_relay.plugin.PluginConfig( + components=[ + nemo_relay.plugin.ComponentSpec( + kind="header-plugin", + config={"header_name": "x-tenant", "value": "tenant-a"}, + ) + ] + ) + report = nemo_relay.plugin.validate(config) + if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + + try: + active_report = await nemo_relay.plugin.initialize(config) + print("Activation report:", active_report) + # Run instrumented application work here. + finally: + nemo_relay.plugin.clear() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + + + +```js +const plugin = require('nemo-relay-node/plugin'); + +const headerPlugin = { + validate(pluginConfig) { + const diagnostics = []; + for (const field of ['header_name', 'value']) { + if (typeof pluginConfig[field] !== 'string') { + diagnostics.push({ + level: 'error', + code: 'header-plugin.invalid_config', + component: 'header-plugin', + field, + message: `${field} must be a string`, + }); + } + } + return diagnostics; + }, + register(pluginConfig, context) { + context.registerLlmRequestIntercept('inject-header', 100, false, ({ request, annotated }) => { + if ( + typeof request !== 'object' || + request === null || + Array.isArray(request) || + typeof request.headers !== 'object' || + request.headers === null || + Array.isArray(request.headers) + ) { + throw new Error('Expected an LLM request object with headers.'); + } + return { + request: { + ...request, + headers: { + ...request.headers, + [String(pluginConfig.header_name)]: String(pluginConfig.value), + }, + }, + annotated, + }; + }); + }, +}; + +void (async () => { + plugin.register('header-plugin', headerPlugin); + + const config = plugin.defaultConfig(); + config.components = [ + plugin.ComponentSpec( + 'header-plugin', + { header_name: 'x-tenant', value: 'tenant-a' }, + { enabled: true }, + ), + ]; + const report = plugin.validate(config); + if (report.diagnostics.some((diagnostic) => diagnostic.level === 'error')) { + throw new Error(JSON.stringify(report.diagnostics)); + } + + try { + const activeReport = await plugin.initialize(config); + console.log('Activation report:', activeReport); + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` + + + +This pattern is useful for: + +- Tenant identity +- Trace correlation +- Region or deployment routing + +## Subscriber Logging Pattern + +Use a subscriber-oriented plugin when the component should watch the full +lifecycle rather than rewrite requests. The following examples log each event +in Python and Node.js. They do not transform events into OpenInference spans. +Refer to +[OpenInference](/configure-plugins/observability/openinference) for the +built-in OpenInference exporter. + + + +```python +import asyncio + +import nemo_relay + +class EventLoggingPlugin: + def register(self, plugin_config, context): + def on_event(event): + print("event", event) + + context.register_subscriber("event-logging", on_event) + +async def main() -> None: + nemo_relay.plugin.register("event-logging", EventLoggingPlugin()) + config = nemo_relay.plugin.PluginConfig( + components=[ + nemo_relay.plugin.ComponentSpec(kind="event-logging", config={}) + ] + ) + report = nemo_relay.plugin.validate(config) + if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + + try: + active_report = await nemo_relay.plugin.initialize(config) + print("Activation report:", active_report) + # Run managed tool or LLM work here to log lifecycle events. + finally: + nemo_relay.plugin.clear() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + + + +```js +const plugin = require('nemo-relay-node/plugin'); + +const eventLoggingPlugin = { + register(pluginConfig, context) { + context.registerSubscriber('event-logging', (event) => { + console.log('event', event); + }); + }, +}; + +void (async () => { + plugin.register('event-logging', eventLoggingPlugin); + const config = plugin.defaultConfig(); + config.components = [plugin.ComponentSpec('event-logging', {})]; + + const report = plugin.validate(config); + if (report.diagnostics.some((diagnostic) => diagnostic.level === 'error')) { + throw new Error(JSON.stringify(report.diagnostics)); + } + + try { + const activeReport = await plugin.initialize(config); + console.log('Activation report:', activeReport); + // Run managed tool or LLM work here to log lifecycle events. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` + + + +This is the right pattern when the component: + +- Logs events across tools and LLMs +- Adds application-specific logging or filtering +- Should not change execution behavior + +## Multi-Surface Policy Bundle + +A plugin can register more than one runtime surface when one configuration document controls a related behavior bundle. + +The following Python example installs a subscriber and an LLM request intercept +from one component configuration: + +```python +import asyncio +from typing import Any + +import nemo_relay + +class CorrelationPolicy: + def validate(self, plugin_config: dict[str, Any]) -> list[dict[str, str]]: + if isinstance(plugin_config.get("header_name"), str): + return [] + return [{ + "level": "error", + "code": "correlation-policy.invalid_config", + "component": "correlation-policy", + "field": "header_name", + "message": "header_name must be a string", + }] + + def register( + self, + plugin_config: dict[str, Any], + context: nemo_relay.plugin.PluginContext, + ) -> None: + def on_event(event: nemo_relay.Event) -> None: + print("event", event) + + def add_correlation_header( + name: str, + request: nemo_relay.LLMRequest, + annotated: nemo_relay.AnnotatedLLMRequest | None, + ) -> nemo_relay.LLMRequestInterceptOutcome: + headers = request.headers.copy() + headers[plugin_config["header_name"]] = "policy-bundle" + return nemo_relay.LLMRequestInterceptOutcome( + nemo_relay.LLMRequest(headers=headers, content=request.content), + annotated, + ) + + context.register_subscriber("event-logging", on_event) + context.register_llm_request_intercept( + "add-correlation-header", 100, False, add_correlation_header + ) + +async def main() -> None: + nemo_relay.plugin.register("correlation-policy", CorrelationPolicy()) + config = nemo_relay.plugin.PluginConfig( + components=[ + nemo_relay.plugin.ComponentSpec( + kind="correlation-policy", + config={"header_name": "x-correlation-source"}, + ) + ] + ) + report = nemo_relay.plugin.validate(config) + if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + + try: + await nemo_relay.plugin.initialize(config) + # Run managed tool or LLM work here. + finally: + nemo_relay.plugin.clear() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +This bundle registers: + +- An event-logging subscriber +- An LLM request intercept that adds correlation metadata + +Use this pattern when one component makes the configured behavior easier to +reason about than several unrelated plugin components. Keep each registered +surface small. Make the component configuration explicit about which surfaces +are enabled. + +## Framework-Neutral Plugin Design + +Plugins can stay framework-agnostic if they operate on the normalized runtime data rather than framework-specific objects. + +Good examples: + +- Rewrite provider headers +- Emit tracing data +- Attach scheduling hints +- Apply cross-framework safety policies diff --git a/docs/build-plugins/language-binding/register-behavior.mdx b/docs/build-plugins/language-binding/register-behavior.mdx new file mode 100644 index 000000000..1bf911723 --- /dev/null +++ b/docs/build-plugins/language-binding/register-behavior.mdx @@ -0,0 +1,380 @@ +--- +title: "Register Plugin Behavior" +description: "Register validated NeMo Relay plugin configuration through PluginContext and manage its lifecycle." +position: 5 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use this guide after you define plugin configuration validation and before the +plugin installs NeMo Relay runtime behavior. + +## What You Build + +Register a plugin kind, initialize validated configuration, install subscribers +or middleware through `PluginContext`, and clear active plugin configuration +during teardown. + +## Use PluginContext + +`PluginContext` is the component-scoped registration surface that Relay passes +to the plugin during initialization. Register subscribers, guardrails, and +intercepts through this context instead of through global registration calls in +application startup. + +The context gives the plugin system three important guarantees: + +- The runtime qualifies names for the component instance. +- Relay rolls back partial setup if one registration fails. +- Plugin diagnostics can identify the affected configured component when the + plugin includes `component` in each diagnostic. + +Use the context only after validation succeeds. Keep validation deterministic and +side-effect free. Inspect configuration and return diagnostics. Create runtime +objects and attach them to the context during registration. + +## Header Plugin Example + +The same model applies in every binding: validate component-local config, then install middleware through the component-scoped registration context. + + + +```python +from typing import Any + +import nemo_relay + +class HeaderPlugin: + def validate(self, plugin_config: dict[str, Any]) -> list[dict[str, str]]: + diagnostics = [] + for field in ("header_name", "value"): + if not isinstance(plugin_config.get(field), str): + diagnostics.append({ + "level": "error", + "code": "header-plugin.invalid_config", + "component": "header-plugin", + "field": field, + "message": f"{field} must be a string", + }) + return diagnostics + + def register(self, plugin_config: dict[str, Any], context: nemo_relay.plugin.PluginContext): + def add_header( + name: str, + request: nemo_relay.LLMRequest, + annotated: nemo_relay.AnnotatedLLMRequest | None + ) -> nemo_relay.LLMRequestInterceptOutcome: + headers = request.headers.copy() + headers[plugin_config["header_name"]] = plugin_config["value"] + return nemo_relay.LLMRequestInterceptOutcome( + nemo_relay.LLMRequest(headers=headers, content=request.content), + annotated, + ) + + context.register_llm_request_intercept("inject-header", 100, False, add_header) + +``` + + + + +```js +const plugin = require('nemo-relay-node/plugin'); + +const headerPlugin = { + validate(pluginConfig) { + const diagnostics = []; + for (const field of ['header_name', 'value']) { + if (typeof pluginConfig[field] !== 'string') { + diagnostics.push({ + level: 'error', + code: 'header-plugin.invalid_config', + component: 'header-plugin', + field, + message: `${field} must be a string`, + }); + } + } + return diagnostics; + }, + register(pluginConfig, context) { + context.registerLlmRequestIntercept('inject-header', 100, false, ({ request, annotated }) => { + if ( + typeof request !== 'object' || + request === null || + Array.isArray(request) || + typeof request.headers !== 'object' || + request.headers === null || + Array.isArray(request.headers) + ) { + throw new Error('Expected an LLM request object with headers.'); + } + return { + request: { + ...request, + headers: { + ...request.headers, + [String(pluginConfig.header_name)]: String(pluginConfig.value), + }, + }, + annotated, + }; + }); + }, +}; + +``` + + + + +```rust +use nemo_relay::api::llm::LlmRequestInterceptOutcome; +use nemo_relay::plugin::{ + ConfigDiagnostic, DiagnosticLevel, Plugin, PluginRegistrationContext, Result as PluginResult, +}; +use serde_json::{Map, Value as Json}; +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; + +struct HeaderPlugin; + +impl Plugin for HeaderPlugin { + fn plugin_kind(&self) -> &str { + "header-plugin" + } + + fn validate(&self, plugin_config: &Map) -> Vec { + let mut diagnostics = Vec::new(); + + for field in ["header_name", "value"] { + match plugin_config.get(field) { + Some(Json::String(_)) => {} + Some(_) => diagnostics.push(ConfigDiagnostic { + level: DiagnosticLevel::Error, + code: "header-plugin.invalid_config".into(), + component: Some("header-plugin".into()), + field: Some(field.into()), + message: format!("{field} must be a string"), + }), + None => diagnostics.push(ConfigDiagnostic { + level: DiagnosticLevel::Error, + code: "header-plugin.invalid_config".into(), + component: Some("header-plugin".into()), + field: Some(field.into()), + message: format!("{field} is required"), + }), + } + } + + diagnostics + } + + fn register<'a>( + &'a self, + plugin_config: &Map, + ctx: &'a mut PluginRegistrationContext, + ) -> Pin> + Send + 'a>> { + let header_name = plugin_config + .get("header_name") + .and_then(Json::as_str) + .unwrap_or("x-plugin") + .to_string(); + let header_value = plugin_config + .get("value") + .and_then(Json::as_str) + .unwrap_or("enabled") + .to_string(); + + Box::pin(async move { + ctx.register_llm_request_intercept( + "inject-header", + 100, + false, + Arc::new(move |_name, mut request, annotated| { + request + .headers + .insert(header_name.clone(), header_value.clone().into()); + Ok(LlmRequestInterceptOutcome::new(request, annotated)) + }), + )?; + Ok(()) + }) + } +} + +``` + + + + + +## Activation APIs + +After you register the plugin kind, use the plugin APIs in this order. Refer to +the [Header Plugin Example](#header-plugin-example) for the registration pattern: + +1. Build a `PluginConfig`. +2. Validate the config. +3. Initialize the config. +4. Inspect the activation report. +5. Clear active config during teardown when needed. + +Register the plugin kind before initialization. With the default +`unknown_component="warn"` policy, `validate()` reports an enabled unregistered +kind as a warning. An error-only check therefore passes, but `initialize()` +raises for an enabled unregistered kind. Register every enabled kind before +initialization, or set +`unknown_component="error"` to make validation fail. + +`validate()` checks only the configuration you pass to it. `initialize()` also +layers discovered `plugins.toml` configuration, so startup can activate or +reject components that a preflight report did not include. Refer to [Plugin +Configuration Files](/configure-plugins/plugin-configuration-files) when file +discovery participates in deployment. + +Append the following entry point to the matching Header Plugin Example above. +Each tab relies on that example's plugin definition and imports. Together, the +two blocks register the custom plugin, validate configuration, initialize it, +inspect the activation report and available kinds, and clear active +configuration: + + + +Append this entry point to the Python Header Plugin Example above. + +```python +import asyncio + +import nemo_relay + +async def main() -> None: + nemo_relay.plugin.register("header-plugin", HeaderPlugin()) + + config = nemo_relay.plugin.PluginConfig() + config.components = [ + nemo_relay.plugin.ComponentSpec( + kind="header-plugin", + config={"header_name": "x-tenant", "value": "tenant-a"}, + ) + ] + + report = nemo_relay.plugin.validate(config) + if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + + try: + active_report = await nemo_relay.plugin.initialize(config) + print("Activation report:", active_report) + print("Available kinds:", nemo_relay.plugin.list_kinds()) + finally: + nemo_relay.plugin.clear() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + + + + +Append this entry point to the Node.js Header Plugin Example above. + +```js +void (async () => { + plugin.register('header-plugin', headerPlugin); + + const config = plugin.defaultConfig(); + config.components = [ + plugin.ComponentSpec( + 'header-plugin', + { header_name: 'x-tenant', value: 'tenant-a' }, + { enabled: true }, + ), + ]; + + const report = plugin.validate(config); + if (report.diagnostics.some((diagnostic) => diagnostic.level === 'error')) { + throw new Error(JSON.stringify(report.diagnostics)); + } + + try { + const activeReport = await plugin.initialize(config); + console.log('Activation report:', activeReport); + console.log('Available kinds:', plugin.listKinds()); + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` + + + + +Append this entry point to the Rust Header Plugin Example above. + +```rust +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, list_plugin_kinds, register_plugin, + validate_plugin_config, PluginComponentSpec, PluginConfig, +}; + +#[tokio::main] +async fn main() -> Result<(), Box> { + register_plugin(Arc::new(HeaderPlugin))?; + + let mut config = PluginConfig::default(); + let mut component = PluginComponentSpec::new("header-plugin"); + component.config.insert("header_name".into(), "x-tenant".into()); + component.config.insert("value".into(), "tenant-a".into()); + config.components.push(component); + + let report = validate_plugin_config(&config); + if report.has_errors() { + return Err(format!("{:?}", report.diagnostics).into()); + } + + let active_report = initialize_plugins(config).await?; + println!("Activation report: {active_report:?}"); + println!("Available kinds: {:?}", list_plugin_kinds()); + clear_plugin_configuration()?; + Ok(()) +} +``` + + + + + +## Registration Checklist + +Before publishing or sharing a plugin: + +1. Validate a correct config and confirm no errors are reported. +2. Validate an intentionally invalid config and confirm diagnostics are actionable. +3. Initialize the plugin and verify the expected subscribers or middleware run. +4. Force one registration failure and confirm partial setup is rolled back. +5. Call `clear()` to remove active component registrations during teardown. + Call `deregister()` too only when a test or embedded runtime must register + the same custom kind again. + +## Common Issues + +Check these symptoms first when the workflow does not behave as expected. + +- **Middleware names collide**: Use component-local names and let the plugin runtime qualify them. +- **Partial registrations remain after failure**: Register through `PluginContext` so rollback can clean up. +- **Registration does validation work**: Move deterministic checks into the validation hook. +- **Global state leaks across component instances**: Create instance-local state during registration or key shared state by component identity. + +## Next Steps + +Use these links to continue from this workflow into the next related task. + +- Add advanced validation and rollout controls with [Design Plugin Configuration](/build-plugins/language-binding/advanced-configuration). +- Review concrete authoring patterns in [Code Examples](/build-plugins/language-binding/code-examples). diff --git a/docs/build-plugins/validate-configuration.mdx b/docs/build-plugins/language-binding/validate-configuration.mdx similarity index 66% rename from docs/build-plugins/validate-configuration.mdx rename to docs/build-plugins/language-binding/validate-configuration.mdx index f420768c8..301eeebe8 100644 --- a/docs/build-plugins/validate-configuration.mdx +++ b/docs/build-plugins/language-binding/validate-configuration.mdx @@ -1,6 +1,6 @@ --- title: "Validate Plugin Configuration" -description: "" +description: "Validate NeMo Relay plugin configuration and return stable diagnostics before activation." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -11,7 +11,9 @@ Use this guide when you have a plugin kind and need predictable diagnostics befo ## What You Build -You will define a JSON-compatible component config, validate required fields and supported values, return structured diagnostics, and confirm that disabled components still report config problems before rollout. +Define a JSON-compatible component configuration, validate required fields and +supported values, return structured diagnostics, and confirm that disabled +components still report configuration problems before rollout. ## Configuration Shape @@ -23,7 +25,15 @@ Each component has: - `enabled`: whether the component should initialize. - `config`: the component-local JSON object passed to validation and registration. -Disabled components are still validated. This lets operators detect config problems before enabling a component in a later rollout. +Relay validates disabled components. This lets operators detect configuration +problems before enabling a component in a later rollout. + +The top-level `policy` controls validation that Relay handles, including unknown +plugin kinds and unsupported document versions. A custom plugin's `validate()` +method controls how its own `config` fields handle unknown fields and +unsupported values. + +The following examples create the same configuration in each supported binding: @@ -49,10 +59,9 @@ config = PluginConfig( -```ts -import type { PluginConfig } from 'nemo-relay-node/plugin'; +```js -const config: PluginConfig = { +const config = { version: 1, components: [ { @@ -91,14 +100,18 @@ let config = PluginConfig { ## Validation Rules -Validation should be deterministic and side-effect free. It should inspect config and return diagnostics; it should not register middleware, open network connections, create clients, or mutate process state. +Keep validation deterministic and side-effect free. Inspect configuration and +return diagnostics. Do not register middleware, open network connections, +create clients, or change process state. Validate these areas first: - Required fields are present. - Field types match the supported shape. -- Unknown fields follow the configured policy. -- Known fields have supported values. +- Your plugin reports unknown component-local fields with the diagnostic level + its contract defines. +- Your plugin reports unsupported component-local values with the diagnostic + level its contract defines. - Cross-field combinations make sense. - Sensitive values are references or secret names, not raw credentials. @@ -127,13 +140,24 @@ Prefer stable diagnostic codes over prose-only messages. The message can improve Use the validation API before initialization and fail deployment if the report contains errors. -This error check does not catch an unknown or unregistered component kind. Under -the default `unknown_component="warn"` policy that case is reported as a warning, -not an error, so the check below passes while `initialize()` still raises for a -kind that is not registered. Register every kind before initialization, or set +This error check does not catch an enabled unknown or unregistered component +kind. Under the default `unknown_component="warn"` policy that case is reported +as a warning, not an error, so the check below passes while `initialize()` still +raises for an enabled kind that is not registered. Register every enabled kind +before initialization, or set `unknown_component="error"` to make validation fail on unknown kinds. +`validate()` checks only the configuration you pass to it. `initialize()` also +layers discovered `plugins.toml` configuration. A preflight report can pass +while the effective startup configuration activates or rejects other +components. Refer to [Plugin Configuration +Files](/configure-plugins/plugin-configuration-files) when file discovery +participates in deployment. + +Append the following validation step to the matching configuration example +above. Each example stops deployment when validation reports an error: + ```python @@ -147,8 +171,8 @@ if has_errors: -```ts -import * as plugin from 'nemo-relay-node/plugin'; +```js +const plugin = require('nemo-relay-node/plugin'); const report = plugin.validate(config); const hasErrors = report.diagnostics.some((diagnostic) => diagnostic.level === 'error'); @@ -164,7 +188,7 @@ use nemo_relay::plugin::validate_plugin_config; let report = validate_plugin_config(&config); if report.has_errors() { - anyhow::bail!("{:?}", report.diagnostics); + panic!("{:?}", report.diagnostics); } ``` @@ -178,7 +202,7 @@ Before sharing a plugin config contract: 1. Validate the smallest correct config. 2. Validate a config with each required field missing. 3. Validate unsupported enum or mode values. -4. Validate unknown fields under each supported policy. +4. Validate unknown component-local fields and their diagnostic levels. 5. Validate disabled components with invalid config. 6. Confirm diagnostics identify the component and field that needs action. @@ -195,6 +219,6 @@ Check these symptoms first when the workflow does not behave as expected. Use these links to continue from this workflow into the next related task. -- Register runtime behavior with [Register Plugin Behavior](/build-plugins/register-behavior). -- Add rollout controls with [Design Plugin Configuration](/build-plugins/advanced-configuration). -- Review concrete validation patterns in [Code Examples](/build-plugins/code-examples). +- Register runtime behavior with [Register Plugin Behavior](/build-plugins/language-binding/register-behavior). +- Add rollout controls with [Design Plugin Configuration](/build-plugins/language-binding/advanced-configuration). +- Review concrete validation patterns in [Code Examples](/build-plugins/language-binding/code-examples). diff --git a/docs/build-plugins/register-behavior.mdx b/docs/build-plugins/register-behavior.mdx deleted file mode 100644 index 82f01803f..000000000 --- a/docs/build-plugins/register-behavior.mdx +++ /dev/null @@ -1,289 +0,0 @@ ---- -title: "Register Plugin Behavior" -description: "" -position: 5 ---- -{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -SPDX-License-Identifier: Apache-2.0 */} - - -Use this guide when plugin config validation is in place and you need the plugin to install real NeMo Relay runtime behavior. - -## What You Build - -You will register a plugin kind, initialize a validated config, install subscribers or middleware through `PluginContext`, and clear active plugin configuration during teardown. - -## Use PluginContext - -`PluginContext` is the component-scoped registration surface passed to the plugin during initialization. Register subscribers, guardrails, and intercepts through this context rather than through global registration calls inside application startup. - -That gives the plugin system three important guarantees: - -- Runtime names can be qualified for the component instance. -- Partial setup can be rolled back if one registration fails. -- Activation reports can identify which configured component installed behavior. - -Use the context only after validation succeeds. Validation should inspect config and return diagnostics; registration should create runtime objects and attach them to the context. - -## Header Plugin Example - -The same model applies in every binding: validate component-local config, then install middleware through the component-scoped registration context. - - - -```python -from typing import Any - -import nemo_relay - -class HeaderPlugin: - def validate(self, plugin_config: dict[str, Any]) -> list[dict[str, str]]: - if "header_name" not in plugin_config or "value" not in plugin_config: - return [{ - "level": "error", - "code": "header-plugin.invalid_config", - "message": "header_name and value are required", - }] - return [] - - def register(self, plugin_config: dict[str, Any], context: nemo_relay.plugin.PluginContext): - def add_header( - name: str, - request: nemo_relay.LLMRequest, - annotated: nemo_relay.AnnotatedLLMRequest | None - ) -> nemo_relay.LLMRequestInterceptOutcome: - headers = request.headers.copy() - headers[plugin_config["header_name"]] = plugin_config["value"] - return nemo_relay.LLMRequestInterceptOutcome( - nemo_relay.LLMRequest(headers=headers, content=request.content), - annotated, - ) - - context.register_llm_request_intercept("inject-header", 100, False, add_header) - -nemo_relay.plugin.register("header-plugin", HeaderPlugin()) -``` - - - - -```ts -import * as plugin from 'nemo-relay-node/plugin'; - -const headerPlugin: plugin.Plugin = { - validate(pluginConfig) { - if (typeof pluginConfig.header_name !== 'string' || typeof pluginConfig.value !== 'string') { - return [{ - level: 'error', - code: 'header-plugin.invalid_config', - message: 'header_name and value are required', - }]; - } - return []; - }, - register(pluginConfig, context) { - context.registerLlmRequestIntercept('inject-header', 100, false, ({ request, annotated }) => ({ - request: { - ...request, - headers: { - ...(request.headers as Record), - [String(pluginConfig.header_name)]: String(pluginConfig.value), - }, - }, - annotated, - })); - }, -}; - -plugin.register('header-plugin', headerPlugin); -``` - - - - -```rust -use nemo_relay::api::llm::LlmRequestInterceptOutcome; -use nemo_relay::plugin::{ - register_plugin, ConfigDiagnostic, DiagnosticLevel, Plugin, PluginRegistrationContext, - Result as PluginResult, -}; -use serde_json::{Map, Value as Json}; -use std::future::Future; -use std::pin::Pin; -use std::sync::Arc; - -struct HeaderPlugin; - -impl Plugin for HeaderPlugin { - fn plugin_kind(&self) -> &str { - "header-plugin" - } - - fn validate(&self, plugin_config: &Map) -> Vec { - let mut diagnostics = Vec::new(); - - for field in ["header_name", "value"] { - match plugin_config.get(field) { - Some(Json::String(_)) => {} - Some(_) => diagnostics.push(ConfigDiagnostic { - level: DiagnosticLevel::Error, - code: "header-plugin.invalid_config".into(), - component: Some("header-plugin".into()), - field: Some(field.into()), - message: format!("{field} must be a string"), - }), - None => diagnostics.push(ConfigDiagnostic { - level: DiagnosticLevel::Error, - code: "header-plugin.invalid_config".into(), - component: Some("header-plugin".into()), - field: Some(field.into()), - message: format!("{field} is required"), - }), - } - } - - diagnostics - } - - fn register<'a>( - &'a self, - plugin_config: &Map, - ctx: &'a mut PluginRegistrationContext, - ) -> Pin> + Send + 'a>> { - let header_name = plugin_config - .get("header_name") - .and_then(Json::as_str) - .unwrap_or("x-plugin") - .to_string(); - let header_value = plugin_config - .get("value") - .and_then(Json::as_str) - .unwrap_or("enabled") - .to_string(); - - Box::pin(async move { - ctx.register_llm_request_intercept( - "inject-header", - 100, - false, - Arc::new(move |_name, mut request, annotated| { - request - .headers - .insert(header_name.clone(), header_value.clone().into()); - Ok(LlmRequestInterceptOutcome::new(request, annotated)) - }), - )?; - Ok(()) - }) - } -} - -register_plugin(Arc::new(HeaderPlugin))?; -``` - - - - - -## Activation APIs - -With the plugin kind registered (see the Header Plugin Example above), use the plugin APIs in this order: - -1. Build a `PluginConfig`. -2. Validate the config. -3. Initialize the config. -4. Inspect the activation report. -5. Clear active config during teardown when needed. - -Register the plugin kind before you initialize. An unregistered kind is reported by `validate()` only as a warning under the default `unknown_component="warn"` policy, so an error-only check still passes, but `initialize()` raises for a kind that was never registered. - - - -```python -import nemo_relay - -config = nemo_relay.plugin.PluginConfig() -config.components = [ - nemo_relay.plugin.ComponentSpec( - kind="header-plugin", - config={"header_name": "x-tenant", "value": "tenant-a"}, - ) -] - -report = nemo_relay.plugin.validate(config) -active_report = await nemo_relay.plugin.initialize(config) -kinds = nemo_relay.plugin.list_kinds() -nemo_relay.plugin.clear() -``` - - - - -```ts -import * as plugin from 'nemo-relay-node/plugin'; - -const config = plugin.defaultConfig(); -config.components = [ - plugin.ComponentSpec( - 'header-plugin', - { header_name: 'x-tenant', value: 'tenant-a' }, - { enabled: true }, - ), -]; - -const report = plugin.validate(config); -const activeReport = await plugin.initialize(config); -const kinds = plugin.listKinds(); -plugin.clear(); -``` - - - - -```rust -use nemo_relay::plugin::{ - clear_plugin_configuration, initialize_plugins, list_plugin_kinds, validate_plugin_config, - PluginComponentSpec, PluginConfig, -}; - -let mut config = PluginConfig::default(); -let mut component = PluginComponentSpec::new("header-plugin"); -component.config.insert("header_name".into(), "x-tenant".into()); -component.config.insert("value".into(), "tenant-a".into()); -config.components.push(component); - -let report = validate_plugin_config(&config); -let active_report = initialize_plugins(config).await?; -let kinds = list_plugin_kinds(); -clear_plugin_configuration()?; -``` - - - - - -## Registration Checklist - -Before publishing or sharing a plugin: - -1. Validate a correct config and confirm no errors are reported. -2. Validate an intentionally invalid config and confirm diagnostics are actionable. -3. Initialize the plugin and verify the expected subscribers or middleware run. -4. Force one registration failure and confirm partial setup is rolled back. -5. Deregister or clear active config during teardown. - -## Common Issues - -Check these symptoms first when the workflow does not behave as expected. - -- **Middleware names collide**: Use component-local names and let the plugin runtime qualify them. -- **Partial registrations remain after failure**: Register through `PluginContext` so rollback can clean up. -- **Registration does validation work**: Move deterministic checks into the validation hook. -- **Global state leaks across component instances**: Create instance-local state during registration or key shared state by component identity. - -## Next Steps - -Use these links to continue from this workflow into the next related task. - -- Add advanced validation and rollout controls with [Design Plugin Configuration](/build-plugins/advanced-configuration). -- Review concrete authoring patterns in [Code Examples](/build-plugins/code-examples). diff --git a/docs/configure-plugins/about.mdx b/docs/configure-plugins/about.mdx new file mode 100644 index 000000000..3612a0b5e --- /dev/null +++ b/docs/configure-plugins/about.mdx @@ -0,0 +1,52 @@ +--- +title: "Configure Plugins" +sidebar-title: "About" +description: "Configure built-in and discoverable NeMo Relay plugins with plugins.toml." +position: 1 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use this section to configure plugin behavior that Relay loads for a process. +The runtime discovers and layers `plugins.toml` for direct Rust, Python, and +Node.js integrations as well as for the `nemo-relay` CLI gateway. + +Choose the path that matches your goal: + +| Goal | Start here | +| --- | --- | +| Configure a built-in Relay component | Choose the component guide below. Use `plugins.toml` or the binding API. | +| Add a plugin someone else packaged | [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) | +| Create a reusable plugin | [Build Plugins](/build-plugins/about) | + +## Built-in Plugins + +Relay ships these first-party components. Configure them as `[[components]]` +entries in `plugins.toml`. + +- [Observability](/configure-plugins/observability/about) exports ATOF, ATIF, + OpenTelemetry, or OpenInference data. +- [Adaptive](/configure-plugins/adaptive/about) configures adaptive runtime + behavior. +- [NeMo Guardrails](/configure-plugins/nemo-guardrails/about) installs + Guardrails-backed policy checks. +- [PII Redaction](/configure-plugins/pii-redaction/about) sanitizes sensitive + data in observability payloads. +- [Model Pricing](/configure-plugins/model-pricing) configures catalog sources + for cost estimates on managed LLM responses. + +## Guides + +Follow the guides below to configure plugins. + +- [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) + explains runtime `plugins.toml` discovery, precedence, merging, and CLI + editing. +- [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) + explains manifest-backed native and worker plugin installation, validation, + trust policy, and activation. + +Use [Build Plugins](/build-plugins/about) only when you are authoring the +plugin itself. It covers generic plugin components, manifests, native ABI +plugins, Python workers, and the `grpc-v1` protocol. diff --git a/docs/adaptive-plugin/about.mdx b/docs/configure-plugins/adaptive/about.mdx similarity index 79% rename from docs/adaptive-plugin/about.mdx rename to docs/configure-plugins/adaptive/about.mdx index b053c8dc9..e6570839d 100644 --- a/docs/adaptive-plugin/about.mdx +++ b/docs/configure-plugins/adaptive/about.mdx @@ -1,7 +1,7 @@ --- title: "Adaptive" sidebar-title: "About" -description: "" +description: "Configure the built-in Adaptive plugin for measured runtime tuning." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -45,13 +45,13 @@ If instrumentation is not in place yet, start with ## Pages -- [Adaptive Configuration](/adaptive-plugin/configuration) documents the full plugin +- [Adaptive Configuration](/configure-plugins/adaptive/configuration) documents the full plugin component shape, validation, activation, teardown, and whole-plugin settings. -- [ACG](/adaptive-plugin/acg) explains Adaptive Cache Governor configuration and what prompt +- [ACG](/configure-plugins/adaptive/acg) explains Adaptive Cache Governor configuration and what prompt cache planning accomplishes. -- [Adaptive Hints](/adaptive-plugin/adaptive-hints) explains request hint injection and how +- [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) explains request hint injection and how downstream model paths can consume the hints. State, telemetry, tool parallelism, and policy are whole-plugin configuration -areas. They are documented on [Adaptive Configuration](/adaptive-plugin/configuration) rather +areas. They are documented on [Adaptive Configuration](/configure-plugins/adaptive/configuration) rather than as separate area pages. diff --git a/docs/adaptive-plugin/acg.mdx b/docs/configure-plugins/adaptive/acg.mdx similarity index 79% rename from docs/adaptive-plugin/acg.mdx rename to docs/configure-plugins/adaptive/acg.mdx index 44783ebc9..a0f5f536b 100644 --- a/docs/adaptive-plugin/acg.mdx +++ b/docs/configure-plugins/adaptive/acg.mdx @@ -1,6 +1,6 @@ --- title: "Adaptive Cache Governor (ACG)" -description: "" +description: "Configure Adaptive Cache Governor for provider-aware prompt-cache planning." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -16,6 +16,8 @@ section is optional. Omit it to keep cache planning disabled. ## `plugins.toml` Example +Add the following Adaptive Cache Governor configuration to `plugins.toml`: + ```toml version = 1 @@ -52,11 +54,19 @@ prompt samples. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the -Adaptive Cache Governor (ACG) runtime lifecycle. +Adaptive Cache Governor (ACG) runtime lifecycle. The following examples +configure and activate ACG through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -76,12 +86,15 @@ report = nemo_relay.plugin.validate(plugin_config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await nemo_relay.plugin.initialize(plugin_config) -try: - # Run instrumented application work here. - pass -finally: - nemo_relay.plugin.clear() +async def main(): + await nemo_relay.plugin.initialize(plugin_config) + try: + # Run instrumented application work here. + pass + finally: + nemo_relay.plugin.clear() + +asyncio.run(main()) ``` @@ -104,19 +117,28 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } -await plugin.initialize(pluginConfig); -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} +void (async () => { + await plugin.initialize(pluginConfig); + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); ``` ```rust -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; -use nemo_relay_adaptive::plugin_component::ComponentSpec; +#[tokio::main] +async fn main() -> Result<(), Box> { +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; +use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{ AcgComponentConfig, AdaptiveConfig, BackendSpec, StateConfig, TelemetryComponentConfig, }; @@ -138,10 +160,17 @@ adaptive.acg = Some(AcgComponentConfig { let mut plugin_config = PluginConfig::default(); plugin_config.components.push(ComponentSpec::new(adaptive).into()); +register_adaptive_component()?; let report = validate_plugin_config(&plugin_config); assert!(!report.has_errors()); -let active = initialize_plugins(plugin_config).await?; +let _active = initialize_plugins(plugin_config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -155,6 +184,8 @@ directly instead of activating the top-level plugin component. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -167,12 +198,12 @@ adaptive_config = nemo_relay.adaptive.AdaptiveConfig( ) runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) -await runtime.register() +asyncio.run(runtime.register()) try: # Run instrumented application work here. runtime.wait_for_idle() finally: - await runtime.shutdown() + asyncio.run(runtime.shutdown()) ``` @@ -183,6 +214,8 @@ Use the Plugin Configuration example above when activating ACG from Node.js. ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay_adaptive::{ AcgComponentConfig, AdaptiveConfig, AdaptiveRuntime, BackendSpec, StateConfig, TelemetryComponentConfig, @@ -209,6 +242,8 @@ runtime.register().await?; runtime.wait_for_idle(); runtime.shutdown().await?; +Ok(()) +} ``` @@ -216,6 +251,8 @@ runtime.shutdown().await?; ## Fields +The following table describes Adaptive Cache Governor settings: + | Field | Default | Notes | |---|---|---| | `provider` | `passthrough` | `passthrough`, `anthropic`, or `openai`. | @@ -240,7 +277,7 @@ Provider-specific cache hints are useful only when the request surface supports them. Validate against representative LLM traffic before enabling ACG in production. -## Common Validation Failures +## Common Configuration and Runtime Issues - `provider` is not one of `passthrough`, `anthropic`, or `openai`. - Stability thresholds are outside the supported numeric range. diff --git a/docs/adaptive-plugin/adaptive-hints.mdx b/docs/configure-plugins/adaptive/adaptive-hints.mdx similarity index 77% rename from docs/adaptive-plugin/adaptive-hints.mdx rename to docs/configure-plugins/adaptive/adaptive-hints.mdx index d2fdec644..4f2530ddf 100644 --- a/docs/adaptive-plugin/adaptive-hints.mdx +++ b/docs/configure-plugins/adaptive/adaptive-hints.mdx @@ -1,6 +1,6 @@ --- title: "Adaptive Hints" -description: "" +description: "Configure Adaptive Hints to add guidance metadata to managed LLM requests." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -11,11 +11,13 @@ Use Adaptive Hints when downstream model calls or provider adapters can safely receive guidance metadata from the adaptive runtime. Adaptive hints register as LLM request intercepts. Lower numeric priority values -run earlier in the intercept chain. The default priority is chosen relative to -other middleware rather than as a standalone importance score. +run earlier in the intercept chain. The plugin sets the default priority +relative to other middleware rather than as a standalone importance score. ## `plugins.toml` Example +Add the following Adaptive Hints configuration to `plugins.toml`: + ```toml version = 1 @@ -47,11 +49,19 @@ allowing later request intercepts to continue running. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the -Adaptive Hints request-intercept lifecycle. +Adaptive Hints request-intercept lifecycle. The following examples configure +and activate Adaptive Hints through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -73,12 +83,15 @@ report = nemo_relay.plugin.validate(plugin_config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await nemo_relay.plugin.initialize(plugin_config) -try: - # Run instrumented application work here. - pass -finally: - nemo_relay.plugin.clear() +async def main(): + await nemo_relay.plugin.initialize(plugin_config) + try: + # Run instrumented application work here. + pass + finally: + nemo_relay.plugin.clear() + +asyncio.run(main()) ``` @@ -103,19 +116,28 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } -await plugin.initialize(pluginConfig); -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} +void (async () => { + await plugin.initialize(pluginConfig); + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); ``` ```rust -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; -use nemo_relay_adaptive::plugin_component::ComponentSpec; +#[tokio::main] +async fn main() -> Result<(), Box> { +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; +use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{ AdaptiveConfig, AdaptiveHintsComponentConfig, BackendSpec, StateConfig, TelemetryComponentConfig, }; @@ -137,10 +159,17 @@ adaptive.adaptive_hints = Some(AdaptiveHintsComponentConfig { let mut plugin_config = PluginConfig::default(); plugin_config.components.push(ComponentSpec::new(adaptive).into()); +register_adaptive_component()?; let report = validate_plugin_config(&plugin_config); assert!(!report.has_errors()); -let active = initialize_plugins(plugin_config).await?; +let _active = initialize_plugins(plugin_config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -154,6 +183,8 @@ directly instead of activating the top-level plugin component. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -168,12 +199,12 @@ adaptive_config = nemo_relay.adaptive.AdaptiveConfig( ) runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) -await runtime.register() +asyncio.run(runtime.register()) try: # Run instrumented application work here. nemo_relay.adaptive.set_latency_sensitivity(8) finally: - await runtime.shutdown() + asyncio.run(runtime.shutdown()) ``` @@ -185,6 +216,8 @@ Hints from Node.js. ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay_adaptive::{ set_latency_sensitivity, AdaptiveConfig, AdaptiveHintsComponentConfig, AdaptiveRuntime, BackendSpec, StateConfig, TelemetryComponentConfig, @@ -211,6 +244,8 @@ runtime.register().await?; set_latency_sensitivity(8).ok(); runtime.shutdown().await?; +Ok(()) +} ``` @@ -218,6 +253,8 @@ runtime.shutdown().await?; ## Fields +The following table describes Adaptive Hints settings: + | Field | Default | Notes | |---|---|---| | `priority` | `100` | Request intercept priority. Lower values run earlier. | @@ -236,7 +273,7 @@ header and body location. The hints do not replace the application callback or change the returned value by themselves. Downstream code must explicitly interpret the metadata before behavior changes. -## Common Validation Failures +## Common Configuration and Runtime Issues - Unknown adaptive hint fields when unknown fields are treated as errors. - `inject_body_path` does not match the request shape expected by downstream diff --git a/docs/adaptive-plugin/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx similarity index 84% rename from docs/adaptive-plugin/configuration.mdx rename to docs/configure-plugins/adaptive/configuration.mdx index 55621aef7..570729b34 100644 --- a/docs/adaptive-plugin/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -1,7 +1,7 @@ --- title: "Adaptive Configuration" sidebar-title: "Configuration" -description: "" +description: "Configure the built-in Adaptive component, including state, telemetry, hints, and ACG." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -16,8 +16,8 @@ Field names stay `snake_case` in every binding and in `plugins.toml`, even when language helper functions use language-native naming conventions. For plugin file discovery, precedence, merge behavior, editor controls, and -gateway conflict rules, see -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +gateway conflict rules, refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ## Component Shape @@ -34,8 +34,8 @@ The top-level adaptive object contains: | `acg` | Adaptive Cache Governor prompt-cache planning. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | -The requested area pages cover [Adaptive Cache Governor (ACG)](/adaptive-plugin/acg) and -[Adaptive Hints](/adaptive-plugin/adaptive-hints). State, telemetry, tool parallelism, and +The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg) and +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints). State, telemetry, tool parallelism, and policy remain whole-plugin settings: - Use `state.backend.kind = "in_memory"` for local experiments. @@ -48,6 +48,8 @@ policy remain whole-plugin settings: ## `plugins.toml` Example +Add the following whole-plugin Adaptive configuration to `plugins.toml`: + ```toml version = 1 @@ -98,9 +100,19 @@ requests can be observed without provider-specific cache translation. ## Per-Language Plugin Configuration +The following examples configure and activate the Adaptive component through a +language binding: + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. + ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -127,7 +139,7 @@ report = nemo_relay.plugin.validate(plugin_config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -active = await nemo_relay.plugin.initialize(plugin_config) +active = asyncio.run(nemo_relay.plugin.initialize(plugin_config)) ``` @@ -157,14 +169,21 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } -const active = await plugin.initialize(pluginConfig); +plugin.initialize(pluginConfig).catch((error) => { + console.error(error); + process.exitCode = 1; +}); ``` ```rust -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; -use nemo_relay_adaptive::plugin_component::ComponentSpec; +#[tokio::main] +async fn main() -> Result<(), Box> { +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; +use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{ AdaptiveConfig, BackendSpec, @@ -197,10 +216,17 @@ adaptive.acg = Some(AcgComponentConfig { let mut plugin_config = PluginConfig::default(); plugin_config.components.push(ComponentSpec::new(adaptive).into()); +register_adaptive_component()?; let report = validate_plugin_config(&plugin_config); assert!(!report.has_errors()); -let active = initialize_plugins(plugin_config).await?; +let _active = initialize_plugins(plugin_config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -214,6 +240,8 @@ directly instead of activating the top-level plugin component. ```python +import asyncio + import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( @@ -233,12 +261,12 @@ adaptive_config = nemo_relay.adaptive.AdaptiveConfig( ) runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) -await runtime.register() +asyncio.run(runtime.register()) try: # Run instrumented application work here. runtime.wait_for_idle() finally: - await runtime.shutdown() + asyncio.run(runtime.shutdown()) ``` @@ -250,6 +278,8 @@ activating adaptive behavior from Node.js. ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay_adaptive::{ AcgComponentConfig, AdaptiveConfig, AdaptiveHintsComponentConfig, AdaptiveRuntime, BackendSpec, StateConfig, TelemetryComponentConfig, ToolParallelismComponentConfig, @@ -281,12 +311,14 @@ runtime.register().await?; runtime.wait_for_idle(); runtime.shutdown().await?; +Ok(()) +} ``` -## Validation And Teardown +## Validation and Teardown Validate plugin configuration before initialization. Disabled adaptive components are still validated, which lets operators prepare a rollout before diff --git a/docs/configure-plugins/discoverable-plugins.mdx b/docs/configure-plugins/discoverable-plugins.mdx new file mode 100644 index 000000000..3f73c4b28 --- /dev/null +++ b/docs/configure-plugins/discoverable-plugins.mdx @@ -0,0 +1,118 @@ +--- +title: "Configure Discoverable Plugins" +description: "Install, validate, trust, and configure manifest-backed NeMo Relay plugins." +position: 3 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use this guide to add a manifest-backed plugin that another author has +packaged. Discoverable plugins are separate from built-in `[[components]]`: +their `relay-plugin.toml` manifest describes a native shared library or a +`grpc-v1` worker, while `plugins.toml` records where Relay can find it and +stores its component configuration. + + +Discoverable plugins are trusted extensions. Native plugins run in the gateway +process. Worker plugins run in a separate process, but process isolation is not +a security sandbox. Install manifests and artifacts only from sources you +trust. + + +## Add and Enable a Plugin + +Validate the manifest before registering it in project configuration: + +```bash +nemo-relay plugins validate ./acme-plugin/relay-plugin.toml +nemo-relay plugins add --project ./acme-plugin/relay-plugin.toml +nemo-relay plugins inspect acme.plugin +nemo-relay plugins enable acme.plugin +nemo-relay plugins validate acme.plugin +``` + +`add` writes a `[[plugins.dynamic]]` reference to the selected `plugins.toml` +scope and stores CLI lifecycle state next to it. `enable` changes that +lifecycle state; it does not load code immediately. Relay validates and loads +enabled plugins when the gateway starts. Use `plugins list`, `plugins inspect`, +and `plugins validate` to review current state and diagnostics. Use `disable` +or `remove` to stop loading a registered plugin. + +You can also add the reference directly when configuration is provisioned by +automation: + +```toml +[[plugins.dynamic]] +manifest = "./acme-plugin/relay-plugin.toml" + +[plugins.dynamic.config] +mode = "audit" +``` + +The manifest path resolves relative to the `plugins.toml` file. The `config` +table supplies the synthesized component configuration. Before enabling or +running a plugin, use `nemo-relay plugins validate ` to check the +manifest, trust evidence, and optional static JSON Schema. During gateway +activation, Relay loads the enabled adapter before it validates the synthesized +component; a worker runs its `Validate` call after its process starts. + +Do not add a Python worker only with this TOML record. Run +`nemo-relay plugins add ` to register a Python worker because Relay must +create and retain its managed Python environment before it can activate the +worker. + +## Validate Before Loading Code + +Relay validates a manifest before it activates plugin code. The manifest must +declare a supported `manifest_version`, plugin ID and kind, Relay compatibility, +the lane-specific ABI or worker protocol, supported capabilities, a disabled +default, and one load contract. A manifest that declares `config_schema` must +also declare the `config_schema` capability. + +Every discoverable plugin needs `source.artifact` and an `integrity.sha256` +digest so Relay can verify the artifact. A host can also require an Ed25519 +signature and trusted public key. Use `nemo-relay plugins validate ` +to evaluate the resolved host policy and artifact trust evidence before you run +the gateway. + +```toml +[plugins.policy.defaults] +startup = "required" +attestation = "signature_required" +trusted_public_keys = ["ed25519:"] +``` + +`startup = "required"` makes lifecycle preflight failures for an enabled plugin +fatal. The default is `optional`, which skips that preflight. A later native or +worker load or activation failure still stops gateway startup; correct or +disable the plugin to start without it. `attestation` accepts `integrity_only`, +`signature_if_present`, or `signature_required`; integrity verification always +checks the declared artifact digest. + +Use `startup = "required"` when failed integrity or signature verification must +prevent worker startup. An optional trust failure records failed lifecycle +state, but does not itself prevent an enabled worker from launching. The native +loader also verifies `load.library`; the worker loader relies on lifecycle trust +evaluation. + +Use `[[plugins.policy.rules]]` to apply an effect by `match_kind` or +`match_plugin_id`, and `[plugins.policy.overrides."plugin.id"]` for one +specific plugin. Rules and overrides can set `allowed`, `startup`, +`attestation`, and `trusted_public_keys`. + +## Select the Authoring Guide + +Ask the plugin author for the correct manifest and artifact. Refer to these +guides when you need to understand the package: + +- [Native Dynamic Plugins (Rust)](/build-plugins/dynamic-plugins/native-dynamic/about) covers + in-process Rust shared libraries and the native ABI. +- [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) + provides the Rust SDK example for a native shared library. +- [gRPC Worker Plugins (Rust)](/build-plugins/dynamic-plugins/grpc-worker/rust/about) covers + Relay-managed Rust workers. +- [gRPC Worker Plugins (Python)](/build-plugins/dynamic-plugins/grpc-worker/python/about) + covers Relay-managed Python workers. +- [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) describes the + shared `grpc-v1` protocol. diff --git a/docs/configure-plugins/model-pricing.mdx b/docs/configure-plugins/model-pricing.mdx new file mode 100644 index 000000000..f4a27d866 --- /dev/null +++ b/docs/configure-plugins/model-pricing.mdx @@ -0,0 +1,94 @@ +--- +title: "Model Pricing" +description: "Configure file and inline pricing catalogs for managed LLM cost estimates." +position: 2 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the built-in `pricing` component to configure model pricing catalogs for +cost estimates on managed LLM responses. Relay leaves cost absent when no +configured source matches the provider, model, and token usage. + +## Configuration + +Add the `pricing` component to `plugins.toml`. The following configuration +loads a JSON catalog and defines an inline catalog. Relay searches sources in +the listed order, so place overrides before their fallbacks: + +```toml +version = 1 + +[[components]] +kind = "pricing" +enabled = true + +[[components.config.sources]] +type = "file" +path = "./pricing.json" + +[[components.config.sources]] +type = "inline" + +[components.config.sources.catalog] +version = 1 + +[[components.config.sources.catalog.entries]] +provider = "openai" +model_id = "gpt-4o-mini" +aliases = ["openai/openai/gpt-4o-mini"] +currency = "USD" +unit = "per_token" +pricing_as_of = "2026-06-06" +pricing_source = "internal-pricing-snapshot" + +[components.config.sources.catalog.entries.rates] +input_per_million = 0.15 +output_per_million = 0.60 +cache_read_per_million = 0.075 + +[components.config.sources.catalog.entries.prompt_cache] +read_accounting = "included_in_prompt_tokens" +``` + +Use `type = "file"` with a JSON catalog path or `type = "inline"` with the +catalog in `plugins.toml`. Relay checks sources and catalog entries in listed +order, and it uses the first entry that matches the provider and model. + +When Relay merges system, project, and user configuration files, it prepends +higher-priority `sources` instead of replacing lower-priority sources. The +effective order is user, project, then system. This lets a narrower user catalog +override a project or enterprise catalog while preserving those catalogs as +fallbacks. + +## Manage Catalog Sources with the CLI + +Run the following commands to validate a catalog and add it to project +configuration: + +```bash +nemo-relay model-pricing validate /path/to/pricing.json +nemo-relay model-pricing init --project +nemo-relay model-pricing add-source /path/to/pricing.json --project +``` + +Use `--user` for the user configuration file or `--global` for +`/etc/nemo-relay/plugins.toml`. `add-source` places a new source ahead of +existing sources by default. Use `--append` to keep it as a fallback. + +## Verify the Active Configuration + +Run the following command to resolve a model against the configured catalogs: + +```bash +nemo-relay model-pricing resolve gpt-4o-mini \ + --provider openai \ + --prompt-tokens 1000 \ + --completion-tokens 500 +``` + +Use `nemo-relay doctor` to validate configured model pricing sources with the +rest of the gateway configuration. Refer to [Add Model Pricing for Cost +Estimates](/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) +for the catalog schema, source precedence, and complete CLI workflow. diff --git a/docs/nemo-guardrails-plugin/about.mdx b/docs/configure-plugins/nemo-guardrails/about.mdx similarity index 95% rename from docs/nemo-guardrails-plugin/about.mdx rename to docs/configure-plugins/nemo-guardrails/about.mdx index 42c3c7398..ca44aebf8 100644 --- a/docs/nemo-guardrails-plugin/about.mdx +++ b/docs/configure-plugins/nemo-guardrails/about.mdx @@ -1,7 +1,7 @@ --- title: "NeMo Guardrails Plugin" sidebar-title: "About" -description: "" +description: "Configure NeMo Guardrails policy around managed LLM and tool execution." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -14,7 +14,7 @@ system. The built-in plugin component has kind `nemo_guardrails` and is available as a first-party NeMo Relay plugin. -The plugin is designed around backend modes: +The plugin supports these backend modes: - `remote` - Calls a Guardrails service over HTTP(S), including streaming over the same @@ -86,10 +86,6 @@ This distinction matters: - Managed surfaces also give NeMo Relay a stable runtime boundary for its own middleware ordering, lifecycle behavior, and observability marks. -In practice: - -- Managed surfaces are places where NeMo Relay is holding the steering wheel. - The forwarded request-default side is more mode-specific: - In `remote` mode, `request_defaults` fields are forwarded to the selected @@ -116,6 +112,6 @@ separate managed middleware surfaces in NeMo Relay. ## Pages -- [NeMo Guardrails Configuration](/nemo-guardrails-plugin/configuration) +- [NeMo Guardrails Configuration](/configure-plugins/nemo-guardrails/configuration) documents the built-in component shape, mode boundaries, and the detailed support matrix. diff --git a/docs/nemo-guardrails-plugin/configuration.mdx b/docs/configure-plugins/nemo-guardrails/configuration.mdx similarity index 81% rename from docs/nemo-guardrails-plugin/configuration.mdx rename to docs/configure-plugins/nemo-guardrails/configuration.mdx index f33b21ea3..d34013a0d 100644 --- a/docs/nemo-guardrails-plugin/configuration.mdx +++ b/docs/configure-plugins/nemo-guardrails/configuration.mdx @@ -1,7 +1,7 @@ --- title: "NeMo Guardrails Configuration" sidebar-title: "Configuration" -description: "" +description: "Configure remote and local NeMo Guardrails backends for managed execution." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -11,8 +11,8 @@ Use this page when you want to configure the built-in NeMo Guardrails plugin component. The component kind is `nemo_guardrails`. For plugin file discovery, precedence, merge behavior, editor controls, and -gateway conflict rules, see -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +gateway conflict rules, refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). NeMo Relay plugin configuration uses the generic plugin document shape, so @@ -46,12 +46,14 @@ At least one managed Guardrails surface must be enabled. ## Backend Support +The following table compares remote and local backend support: + | Area | `remote` | `local` | |---|---|---| | Built-in component kind and config validation | Supported | Supported | | Managed LLM `input` | Supported | Supported | | Managed LLM `output` | Supported | Supported | -| Managed streaming LLM execution | Supported over the remote HTTP(S) contract | Supported; see [Streaming Boundary](#streaming-boundary) | +| Managed streaming LLM execution | Supported over the remote HTTP(S) contract | Supported; refer to [Streaming Boundary](#streaming-boundary) | | Managed `tool_input` | Not supported against the stock Guardrails remote contract | Supported | | Managed `tool_output` | Supported | Supported | | `request_defaults` pass-through | Supported | Not supported | @@ -76,12 +78,24 @@ Guardrails service still owns the actual policy content. In practice, NeMo Relay decides when managed checks run, while the Guardrails config decides what to block, allow, or rewrite. +### Remote Settings + +The remote backend accepts the following settings: + +| Field | Default | Notes | +|---|---|---| +| `remote.endpoint` | Required | Base `http://` or `https://` URL for the reachable Guardrails service. | +| `remote.config_id` | Omitted | One Guardrails configuration identifier. Set this or `remote.config_ids`, but not both. | +| `remote.config_ids` | `[]` | Multiple Guardrails configuration identifiers to combine. Set this or `remote.config_id`, but not both. | +| `remote.headers` | `{}` | Static string headers sent with every remote request. | +| `remote.timeout_millis` | `3000` | Positive request timeout in milliseconds. | + ### `plugins.toml` Example You can write this config directly in `plugins.toml`, or create and edit it through the CLI with `nemo-relay plugins edit`. For plugin file discovery, -precedence, merge behavior, and editor controls, see -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +precedence, merge behavior, and editor controls, refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ```toml version = 1 @@ -238,12 +252,25 @@ The same ownership boundary still applies: - NeMo Relay decides when managed checks run. - Guardrails-native config still decides what to block, allow, or rewrite. +### Local Settings + +The local backend accepts the following settings: + +| Field | Default | Notes | +|---|---|---| +| `config_path` | Omitted | Native Guardrails configuration directory. Set this or `config_yaml`, but not both. | +| `config_yaml` | Omitted | Inline native Guardrails YAML. Set this or `config_path`, but not both. | +| `colang_content` | Omitted | Inline Colang content. Use only with `config_yaml`. | +| `local.python_module` | `nemoguardrails` | Module that the local worker imports. Set a custom module only when the runtime exposes Guardrails through another import path. | +| `local.python_executable` | `NEMO_RELAY_PYTHON`, otherwise `python3` | Python executable that starts the local worker. | +| `local.python_path` | Omitted | Path prepended to the worker subprocess `PYTHONPATH`. | + ### `plugins.toml` Example You can write this config directly in `plugins.toml`, or create and edit it through the CLI with `nemo-relay plugins edit`. For plugin file discovery, -precedence, merge behavior, and editor controls, see -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +precedence, merge behavior, and editor controls, refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ```toml version = 1 @@ -337,18 +364,18 @@ Guardrails calls the main streaming-output switch When `stream_first = true`, the current local mode uses pass-through-first streaming semantics: -- provider chunks can flow to the caller immediately -- Guardrails evaluates the streamed text in parallel -- if Guardrails later blocks the stream, the call fails at that point even - though some chunks may already have been delivered +- Provider chunks can reach the caller immediately. +- Guardrails evaluates the streamed text in parallel. +- If Guardrails later blocks the stream, the call fails after some chunks have + already reached the caller. The current local mode does not support `rails.output.streaming.stream_first = false` yet. That mode would require Guardrails-first chunk reconstruction: -- Guardrails would need to evaluate streamed text before chunks are released to - the caller -- the local backend would then need to convert Guardrails-approved text back - into valid provider-shaped stream chunks +- Guardrails would need to evaluate streamed text before it releases chunks to + the caller. +- The local backend would then need to convert Guardrails-approved text into + valid provider-shaped stream chunks. That guarded-text-to-provider-chunk adapter does not exist yet in the current local backend. diff --git a/docs/observability-plugin/about.mdx b/docs/configure-plugins/observability/about.mdx similarity index 76% rename from docs/observability-plugin/about.mdx rename to docs/configure-plugins/observability/about.mdx index a6a526e22..30d2bc802 100644 --- a/docs/observability-plugin/about.mdx +++ b/docs/configure-plugins/observability/about.mdx @@ -1,7 +1,7 @@ --- title: "Observability" sidebar-title: "About" -description: "" +description: "Configure NeMo Relay observability exporters for events, traces, and trajectories." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -30,11 +30,11 @@ Format (ATIF), OpenTelemetry, or OpenInference. That makes observability downstream from the runtime contract: -- Scopes decide ownership and parentage -- Middleware decides whether execution continues or what payloads are sanitized -- Codecs provide normalized request and response data when available -- Events record the canonical runtime stream -- Subscribers and exporters consume or project that stream +- Scopes decide ownership and parentage. +- Middleware decides whether execution continues or what payloads are sanitized. +- Codecs provide normalized request and response data when available. +- Events record the canonical runtime stream. +- Subscribers and exporters consume or project that stream. Exporters should not redefine execution semantics. They should preserve the meaning of the underlying events while translating them into the shape expected @@ -44,13 +44,14 @@ Configuration and activation failures are setup failures for the observability component. Delivery failures after activation are downstream exporter failures: application work continues, and explicit exporter barriers such as flush or shutdown can report stored delivery problems. Refer to -[Observability Configuration](/observability-plugin/configuration) for the +[Observability Configuration](/configure-plugins/observability/configuration) for the failure table. The first-party plugin component has kind `observability`. It can install: - Agent Trajectory Observability Format (ATOF) JSONL export for raw lifecycle events. -- Agent Trajectory Interchange Format (ATIF) trajectory export for each top-level agent scope. +- Agent Trajectory Interchange Format (ATIF) trajectory export for each top-level + Agent scope or supported coding-agent turn scope. - OpenTelemetry OTLP trace export. - OpenInference-oriented OTLP trace export. @@ -87,17 +88,17 @@ Choose the exporter based on the downstream system: | Need | Use | |---|---| -| Raw canonical event stream | [Agent Trajectory Observability Format (ATOF)](/observability-plugin/atof) | -| Offline analysis, replay, or evaluation trajectories | [Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif) | -| Generic OTLP traces | [OpenTelemetry](/observability-plugin/opentelemetry) | -| OpenInference-oriented agent and LLM spans | [OpenInference](/observability-plugin/openinference) | +| Raw canonical event stream | [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) | +| Offline analysis, replay, or evaluation trajectories | [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) | +| Generic OTLP traces | [OpenTelemetry](/configure-plugins/observability/opentelemetry) | +| OpenInference-oriented agent and LLM spans | [OpenInference](/configure-plugins/observability/openinference) | Start with in-process event inspection before exporting externally. Add sanitize guardrails before exporters receive sensitive payloads. ## What Success Looks Like -Your first exporter path is wired correctly when: +Your first exporter path works correctly when: - The instrumented request still completes successfully. - The chosen exporter receives scope data for that request, plus tool or LLM @@ -108,11 +109,11 @@ If that basic check fails, use the [Trace Incident Runbook](/resources/troubleshooting/trace-incident-runbook) before adding more exporters or extra config layers. -## Correlating Trajectories And Traces +## Correlating Trajectories and Traces When ATIF and trace exporters observe the same NeMo Relay events, they share NeMo Relay UUIDs for cross-format joins. Plugin-managed ATIF uses the top-level -agent scope UUID as the trajectory `session_id`. ATIF step lineage stores the +trajectory root scope UUID as the trajectory `session_id`. ATIF step lineage stores the event UUID as `step.extra.ancestry.function_id` and the parent UUID as `step.extra.ancestry.parent_id`. @@ -124,10 +125,12 @@ are not written into ATIF. ## Pages -- [Observability Configuration](/observability-plugin/configuration) documents the whole plugin +- [Observability Configuration](/configure-plugins/observability/configuration) documents the whole plugin component shape, activation, validation, and teardown. -- [Agent Trajectory Observability Format (ATOF)](/observability-plugin/atof) covers raw JSONL event stream export. -- [Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif) covers per-agent trajectory export. -- [OpenTelemetry](/observability-plugin/opentelemetry) covers generic OTLP trace export. -- [OpenInference](/observability-plugin/openinference) covers OpenInference-oriented OTLP trace +- [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) covers raw JSONL event stream export. +- [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) covers + trajectory export for top-level Agent scopes and supported coding-agent turn + scopes. +- [OpenTelemetry](/configure-plugins/observability/opentelemetry) covers generic OTLP trace export. +- [OpenInference](/configure-plugins/observability/openinference) covers OpenInference-oriented OTLP trace export. diff --git a/docs/observability-plugin/atif.mdx b/docs/configure-plugins/observability/atif.mdx similarity index 66% rename from docs/observability-plugin/atif.mdx rename to docs/configure-plugins/observability/atif.mdx index 847d8c9c6..c3e16c9ff 100644 --- a/docs/observability-plugin/atif.mdx +++ b/docs/configure-plugins/observability/atif.mdx @@ -1,6 +1,6 @@ --- title: "Agent Trajectory Interchange Format (ATIF)" -description: "" +description: "Configure ATIF trajectory export to local files, HTTP endpoints, or S3-compatible storage." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -8,15 +8,18 @@ SPDX-License-Identifier: Apache-2.0 */} Use the `atif` section when you want one Agent Trajectory Interchange Format -(ATIF) trajectory artifact per top-level agent run. +(ATIF) trajectory artifact per top-level Agent scope or supported coding-agent +turn scope. -The plugin-managed ATIF dispatcher watches for direct child scopes with category -`agent`, creates a scope-local exporter for each one, and writes the trajectory -when that agent scope ends. Nested agent scopes remain in the parent -trajectory. +The plugin-managed ATIF dispatcher creates a scope-local exporter for each +top-level Agent scope and each root-child custom scope marked +`nemo_relay_scope_role = "turn"`. It writes the trajectory when that root scope +ends. Nested Agent scopes remain in the parent trajectory. ## `plugins.toml` Example +Add the following ATIF exporter configuration to `plugins.toml`: + ```toml version = 1 @@ -37,10 +40,13 @@ filename_template = "trajectory-{session_id}.json" ``` This configuration writes a trajectory file such as -`logs/trajectory-.json` for each top-level agent scope. +`logs/trajectory-.json` for each top-level Agent scope or supported +coding-agent turn scope. ## Fields +The following table describes the top-level ATIF settings: + | Field | Default | Notes | |---|---|---| | `enabled` | `false` | Must be `true` to write trajectories. | @@ -51,21 +57,22 @@ This configuration writes a trajectory file such as | `extra` | Omitted | Optional ATIF agent metadata. | | `output_directory` | Current working directory | Directory containing trajectory files. Ignored when `storage` is non-empty. | | `filename_template` | `nemo-relay-atif-{session_id}.json` | Must contain `{session_id}`. | -| `storage` | Omitted | Optional list of remote storage destinations. When non-empty, trajectories are uploaded to every configured backend instead of being written locally. See [Remote storage](#remote-storage). | +| `storage` | Omitted | Optional list of remote storage destinations. When non-empty, trajectories are uploaded to every configured backend instead of being written locally. Refer to [Remote Storage](#remote-storage). | -## Remote storage +## Remote Storage -For sandboxed runtimes such as NemoClaw / Omnistation / OpenShell, where local -trace files are not durable across sessions, configure `storage` to ship -completed trajectories directly to object storage. When `storage` is non-empty -the local file write is replaced by uploads to every configured backend; +Use `storage` when local trace files are not durable across sessions, such as in +sandboxed runtimes. When `storage` is non-empty, Relay uploads each completed +trajectory to every configured backend instead of writing a local file. `output_directory` is ignored. Each storage entry is tagged with a `type` discriminator so additional backends can be added without breaking existing configs. S3-compatible object storage and HTTP endpoints are supported. -### S3-compatible storage +### S3-Compatible Storage + +Configure an S3-compatible destination as follows: ```toml [components.config.atif] @@ -78,7 +85,9 @@ bucket = "nemo-relay-traces" key_prefix = "openshell/" ``` -### HTTP endpoint storage +### HTTP Endpoint Storage + +Configure an HTTP destination as follows: ```toml [components.config.atif] @@ -107,6 +116,8 @@ HTTP `2xx` responses are treated as success. Any non-`2xx` response or transport error records the endpoint as unhealthy and skips it for later trajectories. Other configured destinations continue to receive writes. +The following table describes HTTP storage settings: + | Field | Default | Notes | |---|---|---| | `endpoint` | required | Destination `http://` or `https://` URL. | @@ -114,11 +125,11 @@ trajectories. Other configured destinations continue to receive writes. | `header_env` | `{}` | Header names mapped to environment variable names containing secret values. | | `timeout_millis` | `3000` | Per-request timeout in milliseconds. Must be positive. | -### Multiple destinations +### Multiple Destinations -Add additional `[[components.config.atif.storage]]` tables to fan out the same -trajectory to every destination — for example, an in-cluster MinIO target and a -remote HTTP endpoint: +Add another `[[components.config.atif.storage]]` table to send each trajectory +to every configured destination. For example, this configuration uses an +in-cluster MinIO target and a remote HTTP endpoint: ```toml [components.config.atif] @@ -135,13 +146,14 @@ type = "http" endpoint = "https://observability.example.com/atif" ``` -#### Connection fields +#### Connection Fields + +By default, Relay reads credentials, region, and endpoint URL from standard AWS +environment variables. You can also set non-secret connection fields directly +in the configuration. This lets one file describe multiple S3-compatible +destinations, such as an in-cluster MinIO target and a remote AWS target: -By default, credentials, region, and endpoint URL are read from the standard -AWS environment variables. Any non-secret connection field can also be set -directly in the config so a single file can describe a complete destination, -which is what unblocks running multiple S3-compatible endpoints side by side -(for example, an in-cluster MinIO target and a remote AWS target): +The following table describes S3-compatible connection settings: | Field | Default | Notes | |---|---|---| @@ -155,11 +167,13 @@ which is what unblocks running multiple S3-compatible endpoints side by side Explicit fields take precedence; anything left unset falls back to the matching `AWS_*` environment variable. -#### Secret credential fields +#### Secret Credential Fields Secret values stay out of checked-in config files. Each secret field carries a `_var` suffix and holds the *name* of an environment variable that contains the -secret value. The name is validated at plugin initialization time: +secret value. The plugin validates the name during initialization: + +The following table describes secret credential settings: | Field | Env Var Fallback | Notes | |---|---|---| @@ -167,16 +181,16 @@ secret value. The name is validated at plugin initialization time: | `session_token_var` | `AWS_SESSION_TOKEN` | Name of the env var that holds the optional STS session token. | -Each trajectory is uploaded under `{key_prefix}{rendered_filename}`. The -`filename_template` is rendered the same way it would be for local files, so a -local→remote transition keeps object names stable. A trailing `/` is added to +Relay uploads each trajectory under `{key_prefix}{rendered_filename}`. It +renders `filename_template` the same way it does for local files, so a +local→remote transition keeps object names stable. Relay adds a trailing `/` to `key_prefix` when one is missing. -If an upload fails for a given destination, that destination is recorded as -unhealthy and skipped on later trajectories. The other destinations continue -to receive writes. Fatal dispatcher failures, such as trajectory serialization -failures, stop later ATIF observation and are reported during plugin teardown; -per-destination sink failures are isolated to the failed sink. +If an upload fails for a destination, the ATIF exporter records that destination +as unhealthy and skips it for later trajectories. Other destinations continue +to receive writes. The exporter reports fatal dispatcher failures, such as +trajectory serialization failures, during plugin teardown. It isolates +per-destination sink failures to the failed sink. ## Expected Output @@ -189,13 +203,13 @@ and referenced from parent observation results with child trajectory by ID so consumers can validate the parent and child as one single-file ATIF v1.7 artifact. -The plugin writes each trajectory when the top-level agent scope closes. If the -plugin is cleared while an agent is still open, teardown flushes the partial -trajectory. +The plugin writes each trajectory when its top-level Agent scope or supported +coding-agent turn scope closes. If the plugin is cleared while that root scope +is still open, teardown flushes the partial trajectory. To correlate ATIF with OpenTelemetry or OpenInference traces from the same run, -join on NeMo Relay UUIDs. The plugin-managed ATIF `session_id` is the top-level -agent scope UUID. Each step's `extra.ancestry.function_id` is the event UUID, +join on NeMo Relay UUIDs. The plugin-managed ATIF `session_id` is the +top-level trajectory root scope UUID. Each step's `extra.ancestry.function_id` is the event UUID, and `extra.ancestry.parent_id` is the parent event UUID. Trace spans expose the same values as `nemo_relay.uuid` and `nemo_relay.parent_uuid` attributes. @@ -214,11 +228,19 @@ middleware ordering, or provider payload decoding. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the ATIF -dispatcher lifecycle. +dispatcher lifecycle. The following examples configure and activate the ATIF +exporter through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + from nemo_relay import plugin from nemo_relay.observability import AtifConfig, ComponentSpec, ObservabilityConfig @@ -243,12 +265,15 @@ report = plugin.validate(config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await plugin.initialize(config) -try: - # Run instrumented application work here. - pass -finally: - plugin.clear() +async def main(): + await plugin.initialize(config) + try: + # Run instrumented application work here. + pass + finally: + plugin.clear() + +asyncio.run(main()) ``` @@ -258,7 +283,8 @@ finally: const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); -await plugin.initialize({ +void (async () => { + await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ @@ -273,23 +299,31 @@ await plugin.initialize({ }), }), ], + }); + + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; }); - -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} ``` ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay::observability::plugin_component::{ AtifSectionConfig, ComponentSpec, ObservabilityConfig, }; -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; let component = ComponentSpec::new(ObservabilityConfig { atif: Some(AtifSectionConfig { @@ -313,7 +347,13 @@ let config = PluginConfig { let report = validate_plugin_config(&config); assert!(!report.has_errors()); -let active = initialize_plugins(config).await?; +let _active = initialize_plugins(config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -367,40 +407,43 @@ try { use nemo_relay::api::subscriber::{deregister_subscriber, flush_subscribers, register_subscriber}; use nemo_relay::observability::atif::{AtifAgentInfo, AtifExporter}; -let exporter = AtifExporter::new( - "session-1".to_string(), - AtifAgentInfo { - name: "agent".to_string(), - version: "1.0.0".to_string(), - model_name: Some("demo-model".to_string()), - tool_definitions: None, - extra: None, - }, -); -register_subscriber("atif-exporter", exporter.subscriber())?; - -// Run instrumented application work here. - -flush_subscribers()?; -let trajectory = exporter.export()?; -let trajectory_json = serde_json::to_string_pretty(&trajectory)?; -println!("{trajectory_json}"); - -let _ = deregister_subscriber("atif-exporter")?; -exporter.clear(); +fn main() -> Result<(), Box> { + let exporter = AtifExporter::new( + "session-1".to_string(), + AtifAgentInfo { + name: "agent".to_string(), + version: "1.0.0".to_string(), + model_name: Some("demo-model".to_string()), + tool_definitions: None, + extra: None, + }, + ); + register_subscriber("atif-exporter", exporter.subscriber())?; + + // Run instrumented application work here. + + flush_subscribers()?; + let trajectory = exporter.export()?; + let trajectory_json = serde_json::to_string_pretty(&trajectory)?; + println!("{trajectory_json}"); + + let _ = deregister_subscriber("atif-exporter")?; + exporter.clear(); + Ok(()) +} ``` -## Common Validation Failures +## Common Configuration and Runtime Issues - `filename_template` does not contain `{session_id}`. - The output directory is not writable at runtime. - Tool definitions or `extra` metadata are not JSON-compatible. -- The application never opens a top-level `agent` scope, so no trajectory file - is created. +- The application never opens a top-level Agent scope or a supported + coding-agent turn scope, so no trajectory file is created. - `storage[i].type` is unknown or `storage[i].bucket` is empty for some entry. - `storage` is non-empty in a build that was compiled without the `atif-storage` feature. diff --git a/docs/observability-plugin/atof.mdx b/docs/configure-plugins/observability/atof.mdx similarity index 73% rename from docs/observability-plugin/atof.mdx rename to docs/configure-plugins/observability/atof.mdx index dfa2d0103..d0103354e 100644 --- a/docs/observability-plugin/atof.mdx +++ b/docs/configure-plugins/observability/atof.mdx @@ -1,6 +1,6 @@ --- title: "Agent Trajectory Observability Format (ATOF)" -description: "" +description: "Configure raw ATOF event export to JSONL files and streaming endpoints." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -18,6 +18,8 @@ event shape in near real time. ## `plugins.toml` Example +Add the following ATOF exporter configuration to `plugins.toml`: + ```toml version = 1 @@ -38,6 +40,7 @@ mode = "overwrite" url = "http://localhost:8080/events" transport = "http_post" timeout_millis = 3000 +field_name_policy = "replace_dots" ``` This configuration registers the plugin-managed ATOF exporter and writes one @@ -46,6 +49,8 @@ ATOF event to the configured endpoint. ## Fields +The following table describes the top-level ATOF settings: + | Field | Default | Notes | |---|---|---| | `enabled` | `false` | Must be `true` to write events. | @@ -60,12 +65,21 @@ Each endpoint receives the same raw ATOF JSON object that the file exporter writes as one JSONL line. Endpoints are independent: a failed endpoint is skipped or retried without blocking file output or other endpoints. +The following table describes each streaming endpoint: + | Field | Default | Notes | |---|---|---| | `url` | Required | Endpoint URL. | | `transport` | `http_post` | `http_post`, `websocket`, or `ndjson`. | | `headers` | `{}` | String-to-string headers for requests or handshakes. | | `timeout_millis` | `3000` | Per-endpoint timeout. Must be greater than `0`. | +| `field_name_policy` | `preserve` | Field-name handling before Relay sends endpoint events. Accepted values are `preserve` and `replace_dots`. | + +`preserve` sends canonical ATOF field names unchanged. `replace_dots` replaces +dots in JSON object keys with underscores recursively. If replacement produces +a collision, Relay keeps both values and deterministically appends `_2`, `_3`, +and later suffixes as needed. Validation rejects any other +`endpoints[i].field_name_policy` value. - `http_post` sends each event as one JSONL record in an HTTP `POST` request with `Content-Type: application/x-ndjson`. Any `2xx` response is treated as @@ -82,7 +96,7 @@ work, closes streaming connections, and makes later events no-op. ## Expected Output Each emitted scope, tool, LLM, middleware, or mark event is written as one ATOF -JSON object per line. For event field semantics, see +JSON object per line. For event field semantics, refer to [Events](/about-nemo-relay/concepts/events). ATOF is the raw-event export path. It preserves the canonical event shape; it @@ -95,11 +109,19 @@ shutdown so file handles flush. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the ATOF -exporter lifecycle. +exporter lifecycle. The following examples configure and activate the ATOF +exporter through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + from nemo_relay import plugin from nemo_relay.observability import ( AtofConfig, @@ -133,12 +155,15 @@ report = plugin.validate(config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await plugin.initialize(config) -try: - # Run instrumented application work here. - pass -finally: - plugin.clear() +async def main(): + await plugin.initialize(config) + try: + # Run instrumented application work here. + pass + finally: + plugin.clear() + +asyncio.run(main()) ``` @@ -148,7 +173,8 @@ finally: const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); -await plugin.initialize({ +void (async () => { + await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ @@ -167,23 +193,31 @@ await plugin.initialize({ }), }), ], + }); + + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; }); - -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} ``` ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay::observability::plugin_component::{ AtofEndpointSectionConfig, AtofSectionConfig, ComponentSpec, ObservabilityConfig, }; -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; let component = ComponentSpec::new(ObservabilityConfig { atof: Some(AtofSectionConfig { @@ -196,6 +230,7 @@ let component = ComponentSpec::new(ObservabilityConfig { transport: "http_post".into(), headers: Default::default(), timeout_millis: 3000, + field_name_policy: "replace_dots".into(), }], }), ..ObservabilityConfig::default() @@ -210,7 +245,13 @@ let config = PluginConfig { let report = validate_plugin_config(&config); assert!(!report.has_errors()); -let active = initialize_plugins(config).await?; +let _active = initialize_plugins(config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -277,29 +318,32 @@ use nemo_relay::observability::atof::{ AtofEndpointConfig, AtofEndpointTransport, AtofExporter, AtofExporterConfig, AtofExporterMode, }; -let config = AtofExporterConfig::new() - .with_output_directory("logs") - .with_filename("events.jsonl") - .with_mode(AtofExporterMode::Overwrite) - .with_endpoint(AtofEndpointConfig::new( - "http://localhost:8080/events", - AtofEndpointTransport::HttpPost, - )); -let exporter = AtofExporter::new(config)?; -exporter.register("atof-exporter")?; - -// Run instrumented application work here. - -exporter.force_flush()?; -let _ = exporter.deregister("atof-exporter")?; -exporter.shutdown()?; +fn main() -> Result<(), Box> { + let config = AtofExporterConfig::new() + .with_output_directory("logs") + .with_filename("events.jsonl") + .with_mode(AtofExporterMode::Overwrite) + .with_endpoint(AtofEndpointConfig::new( + "http://localhost:8080/events", + AtofEndpointTransport::HttpPost, + )); + let exporter = AtofExporter::new(config)?; + exporter.register("atof-exporter")?; + + // Run instrumented application work here. + + exporter.force_flush()?; + let _ = exporter.deregister("atof-exporter")?; + exporter.shutdown()?; + Ok(()) +} ``` -## Common Validation Failures +## Common Configuration and Runtime Issues - `mode` is not `append` or `overwrite`. - `endpoints[i].url` is empty, `endpoints[i].transport` is not supported, or diff --git a/docs/observability-plugin/configuration.mdx b/docs/configure-plugins/observability/configuration.mdx similarity index 86% rename from docs/observability-plugin/configuration.mdx rename to docs/configure-plugins/observability/configuration.mdx index 447754723..bfc0c1625 100644 --- a/docs/observability-plugin/configuration.mdx +++ b/docs/configure-plugins/observability/configuration.mdx @@ -1,7 +1,7 @@ --- title: "Observability Configuration" sidebar-title: "Configuration" -description: "" +description: "Configure ATOF, ATIF, OpenTelemetry, and OpenInference exporters together." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -12,13 +12,13 @@ Use this page when an application should install standard observability exporters from one plugin configuration document instead of manually registering each subscriber. -The plugin kind is `observability`. It is registered by the core runtime, so +The plugin kind is `observability`. The core runtime registers it, so applications do not need to register a plugin implementation before validation or initialization. For plugin file discovery, precedence, merge behavior, editor controls, and -gateway conflict rules, see -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +gateway conflict rules, refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). Observability plugin configuration uses the generic NeMo Relay plugin document @@ -31,12 +31,14 @@ Node-native `camelCase` option names outside the plugin system. ## What It Installs Every exporter section is optional and defaults to disabled. A section is active -only when it includes `enabled: true`. +only when it includes `enabled = true`. + +The following table lists the runtime behavior that each section installs: | Section | Runtime behavior | |---|---| | `atof` | Registers a global Agent Trajectory Observability Format (ATOF) JSONL exporter for raw lifecycle events. | -| `atif` | Registers one Agent Trajectory Interchange Format (ATIF) dispatcher that writes one trajectory file for each top-level agent scope. | +| `atif` | Registers one Agent Trajectory Interchange Format (ATIF) dispatcher that writes one trajectory file for each top-level Agent scope or supported coding-agent turn scope. | | `opentelemetry` | Registers a global OpenTelemetry OTLP subscriber. | | `openinference` | Registers a global OpenInference OTLP subscriber. | @@ -46,12 +48,14 @@ plugin namespace: - Agent Trajectory Observability Format (ATOF): `__nemo_relay_plugin__observability__atof` - Agent Trajectory Interchange Format (ATIF) dispatcher: `__nemo_relay_plugin__observability__atif` -- Per-agent ATIF scope subscriber: `__nemo_relay_plugin__observability__atif-{agent_scope_uuid}` +- Per-trajectory-root ATIF scope subscriber: `__nemo_relay_plugin__observability__atif-{root_scope_uuid}` - OpenTelemetry: `__nemo_relay_plugin__observability__opentelemetry` - OpenInference: `__nemo_relay_plugin__observability__openinference` ## `plugins.toml` Example +Add the following observability component configuration to `plugins.toml`: + ```toml version = 1 @@ -134,21 +138,23 @@ new plugin configuration becomes active. Runtime delivery failures are fail-open for application work: the tool, LLM, or agent run continues while the affected exporter records, logs, or reports the delivery problem. +The following table describes how each failure affects application work: + | Failure | Behavior | |---|---| | Invalid `plugins.toml`, duplicate component kinds, malformed component shapes, unsupported values, unavailable exporter features, ATOF file-open failures, invalid ATOF endpoint config, unavailable ATOF streaming support, or ATOF endpoint worker startup failures | Validation or initialization fails. If a previous plugin configuration was active, NeMo Relay attempts to restore it after a failed replacement. | | ATOF event serialization or file write/flush failure after activation | Application work continues. The exporter stores the failure, stops accepting later events for that file, and returns the stored error from `force_flush()` or `shutdown()`. | | ATOF streaming endpoint connection or send failure after activation | File output and other already-started endpoints continue. Endpoint failures are logged with the endpoint index; endpoint flush and close timeouts are logged instead of blocking shutdown indefinitely. | -| ATIF local file write, HTTP storage, or S3-compatible storage failure | Application work continues. The failed sink is recorded as unhealthy and skipped for later trajectories. Other configured sinks continue to receive writes. | +| ATIF local file write, HTTP storage, or S3-compatible storage failure | Application work continues. The ATIF exporter records the failed sink as unhealthy and skips it for later trajectories. Other configured sinks continue to receive writes. | | ATIF dispatcher serialization or subscriber-management failure | The ATIF dispatcher records a fatal exporter error and stops observing later ATIF events. Other observability sections continue to run. | | OpenTelemetry or OpenInference construction failure | Plugin initialization fails before the subscriber is registered. | | OpenTelemetry or OpenInference export failure after registration | Application work continues. The OTLP exporter reports failures through its runtime logging and flush or shutdown path. | Missing or delayed telemetry is represented as absence of exporter output, not as synthetic success or failure events. NeMo Relay does not backfill events for -subscribers that register late. If the plugin is cleared while an agent scope -is still open, the ATIF dispatcher writes the partial trajectory it has already -observed. +subscribers that register late. If the plugin is cleared while an Agent scope +or supported coding-agent turn scope is still open, the ATIF dispatcher writes +the partial trajectory it has already observed. Use `nemo-relay doctor` to validate local ATOF and ATIF output directories, OTLP HTTP endpoints, and ATOF streaming endpoints. Validate ATIF remote storage @@ -160,6 +166,14 @@ create the output directory and that teardown calls `plugin.clear()` or ## Per-Language Plugin Configuration +The following examples configure and activate the Observability component +through each supported language binding: + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. + ```python @@ -231,7 +245,8 @@ asyncio.run(arun()) const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); -await plugin.initialize({ +void (async () => { + await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ @@ -271,24 +286,32 @@ await plugin.initialize({ }), }), ], + }); + + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; }); - -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} ``` ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay::observability::plugin_component::{ AtifSectionConfig, AtofEndpointSectionConfig, AtofSectionConfig, ComponentSpec, ObservabilityConfig, OtlpSectionConfig, }; -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; let component = ComponentSpec::new(ObservabilityConfig { atof: Some(AtofSectionConfig { @@ -301,6 +324,7 @@ let component = ComponentSpec::new(ObservabilityConfig { transport: "http_post".into(), headers: [("authorization".into(), "Bearer ".into())].into(), timeout_millis: 3000, + field_name_policy: "preserve".into(), }], }), atif: Some(AtifSectionConfig { @@ -341,14 +365,20 @@ let config = PluginConfig { let report = validate_plugin_config(&config); assert!(!report.has_errors()); -let active = initialize_plugins(config).await?; +let _active = initialize_plugins(config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` -## Validation And Teardown +## Validation and Teardown Validate plugin configuration before activating it. The plugin reports unsupported transports, unsupported ATOF modes, invalid ATOF streaming endpoint diff --git a/docs/observability-plugin/openinference.mdx b/docs/configure-plugins/observability/openinference.mdx similarity index 75% rename from docs/observability-plugin/openinference.mdx rename to docs/configure-plugins/observability/openinference.mdx index 7457a1581..56f3e8edc 100644 --- a/docs/observability-plugin/openinference.mdx +++ b/docs/configure-plugins/observability/openinference.mdx @@ -1,6 +1,6 @@ --- title: "OpenInference" -description: "" +description: "Configure OpenInference-oriented OTLP tracing for NeMo Relay events." position: 6 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -11,16 +11,18 @@ Use the `openinference` section when you want NeMo Relay lifecycle events exported as OTLP trace spans with OpenInference-oriented semantics. OpenInference export maps model-centric payloads directly into trace -attributes. Scope, tool, and LLM start inputs become `input.value`; end outputs -become `output.value`; LLM usage metadata maps to token-count attributes when -the provider response includes usage information. For LLM spans, NeMo Relay -emits flattened request and response message attributes from typed codec -annotations and supported replay request/response payloads. Typed codec -annotations also provide tool schema, finish-reason, and invocation-parameter -attributes when those fields are available. +attributes. Scope, tool, and LLM start inputs become `input.value`. End outputs +become `output.value`. LLM usage metadata maps to token-count attributes when +provider responses include usage information. For LLM spans, NeMo Relay emits +flattened request and response message attributes from typed codec annotations +and supported replay request/response payloads. Typed codec annotations also +provide tool schema, finish-reason, and invocation-parameter attributes when +those fields are available. ## `plugins.toml` Example +Add the following OpenInference exporter configuration to `plugins.toml`: + ```toml version = 1 @@ -54,7 +56,7 @@ OpenInference-style OTLP spans to Phoenix or another compatible backend. ## Fields OpenInference uses the same OTLP section shape as -[OpenTelemetry](/observability-plugin/opentelemetry): +[OpenTelemetry](/configure-plugins/observability/opentelemetry): | Field | Default | Notes | |---|---|---| @@ -84,12 +86,12 @@ transport metadata. Each lifecycle span includes `nemo_relay.uuid` and `nemo_relay.parent_uuid` attributes. These values match ATIF `step.extra.ancestry.function_id` and `step.extra.ancestry.parent_id` for the same events. For plugin-managed ATIF, -the root agent span's `nemo_relay.uuid` also matches the ATIF `session_id`. +the trajectory-root span's `nemo_relay.uuid` also matches the ATIF `session_id`. Backend-native `trace_id` and `span_id` values are not written into ATIF. LLM token counts appear as `llm.token_count.prompt`, `llm.token_count.completion`, -`llm.token_count.total`, and `llm.token_count.prompt_details.cache_read`/`cache_write`; -cost appears as USD-denominated `llm.cost.total`. Refer to +`llm.token_count.total`, and `llm.token_count.prompt_details.cache_read`/`cache_write`. +Cost appears as USD-denominated `llm.cost.total`. Refer to [Token and Cost Field Semantics](/integrate-into-frameworks/provider-response-codecs#token-and-cost-field-semantics) for the full mapping. @@ -99,11 +101,19 @@ export. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the -OpenInference subscriber lifecycle. +OpenInference subscriber lifecycle. The following examples configure and +activate the OpenInference exporter through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + from nemo_relay import plugin from nemo_relay.observability import ComponentSpec, ObservabilityConfig, OtlpConfig @@ -131,9 +141,15 @@ report = plugin.validate(config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -async with plugin.plugin(config): - # Run instrumented application work here. - pass +async def main(): + await plugin.initialize(config) + try: + # Run instrumented application work here. + pass + finally: + plugin.clear() + +asyncio.run(main()) ``` @@ -143,7 +159,8 @@ async with plugin.plugin(config): const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); -await plugin.initialize({ +void (async () => { + await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ @@ -165,23 +182,31 @@ await plugin.initialize({ }), }), ], + }); + + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; }); - -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} ``` ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay::observability::plugin_component::{ ComponentSpec, ObservabilityConfig, OtlpSectionConfig, }; -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; let component = ComponentSpec::new(ObservabilityConfig { openinference: Some(OtlpSectionConfig { @@ -208,7 +233,13 @@ let config = PluginConfig { let report = validate_plugin_config(&config); assert!(!report.has_errors()); -let active = initialize_plugins(config).await?; +let _active = initialize_plugins(config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -275,25 +306,28 @@ use nemo_relay::observability::openinference::{ OpenInferenceConfig, OpenInferenceSubscriber, }; -let config = OpenInferenceConfig::new() - .with_service_name("agent-service") - .with_endpoint("http://localhost:6006/v1/traces") - .with_resource_attribute("deployment.environment", "dev"); -let subscriber = OpenInferenceSubscriber::new(config)?; -subscriber.register("openinference-exporter")?; +fn main() -> Result<(), Box> { + let config = OpenInferenceConfig::new() + .with_service_name("agent-service") + .with_endpoint("http://localhost:6006/v1/traces") + .with_resource_attribute("deployment.environment", "dev"); + let subscriber = OpenInferenceSubscriber::new(config)?; + subscriber.register("openinference-exporter")?; -// Run instrumented application work here. + // Run instrumented application work here. -subscriber.force_flush()?; -let _ = subscriber.deregister("openinference-exporter")?; -subscriber.shutdown()?; + subscriber.force_flush()?; + let _ = subscriber.deregister("openinference-exporter")?; + subscriber.shutdown()?; + Ok(()) +} ``` -## Common Validation Failures +## Common Configuration and Runtime Issues - `transport` is not `http_binary` or `grpc`. - Headers or resource attributes are not string-to-string maps. diff --git a/docs/observability-plugin/opentelemetry.mdx b/docs/configure-plugins/observability/opentelemetry.mdx similarity index 78% rename from docs/observability-plugin/opentelemetry.mdx rename to docs/configure-plugins/observability/opentelemetry.mdx index 6bfe6fa0a..70606a24d 100644 --- a/docs/observability-plugin/opentelemetry.mdx +++ b/docs/configure-plugins/observability/opentelemetry.mdx @@ -1,6 +1,6 @@ --- title: "OpenTelemetry" -description: "" +description: "Configure OpenTelemetry Protocol trace export for NeMo Relay events." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -16,6 +16,8 @@ appear in the same tracing pipeline as the rest of the application. ## `plugins.toml` Example +Add the following OpenTelemetry exporter configuration to `plugins.toml`: + ```toml version = 1 @@ -48,6 +50,8 @@ NeMo Relay trace spans to the configured OTLP endpoint. ## Fields +The following table describes OpenTelemetry exporter settings: + | Field | Default | Notes | |---|---|---| | `enabled` | `false` | Must be `true` to construct and register the subscriber. | @@ -70,7 +74,7 @@ root scope. Each lifecycle span includes `nemo_relay.uuid` and `nemo_relay.parent_uuid` attributes. These values match ATIF `step.extra.ancestry.function_id` and `step.extra.ancestry.parent_id` for the same events. For plugin-managed ATIF, -the root agent span's `nemo_relay.uuid` also matches the ATIF `session_id`. +the trajectory-root span's `nemo_relay.uuid` also matches the ATIF `session_id`. Backend-native `trace_id` and `span_id` values are not written into ATIF. For LLM end spans, cost is emitted as `nemo_relay.llm.cost.total` and @@ -86,11 +90,19 @@ graceful shutdown. ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the -OpenTelemetry subscriber lifecycle. +OpenTelemetry subscriber lifecycle. The following examples configure and +activate the OpenTelemetry exporter through each supported language binding. + +`validate()` checks only the supplied in-memory object. `initialize()` also +layers discovered `plugins.toml` configuration. For effective file-backed +validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) +and run the gateway with the same configuration path that production uses. ```python +import asyncio + from nemo_relay import plugin from nemo_relay.observability import ComponentSpec, ObservabilityConfig, OtlpConfig @@ -118,12 +130,15 @@ report = plugin.validate(config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) -await plugin.initialize(config) -try: - # Run instrumented application work here. - pass -finally: - plugin.clear() +async def main(): + await plugin.initialize(config) + try: + # Run instrumented application work here. + pass + finally: + plugin.clear() + +asyncio.run(main()) ``` @@ -133,7 +148,8 @@ finally: const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); -await plugin.initialize({ +void (async () => { + await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ @@ -155,23 +171,31 @@ await plugin.initialize({ }), }), ], + }); + + try { + // Run instrumented application work here. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; }); - -try { - // Run instrumented application work here. -} finally { - plugin.clear(); -} ``` ```rust +#[tokio::main] +async fn main() -> Result<(), Box> { use nemo_relay::observability::plugin_component::{ ComponentSpec, ObservabilityConfig, OtlpSectionConfig, }; -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; let component = ComponentSpec::new(ObservabilityConfig { opentelemetry: Some(OtlpSectionConfig { @@ -198,7 +222,13 @@ let config = PluginConfig { let report = validate_plugin_config(&config); assert!(!report.has_errors()); -let active = initialize_plugins(config).await?; +let _active = initialize_plugins(config).await?; + +// Run instrumented application work here. + +clear_plugin_configuration()?; +Ok(()) +} ``` @@ -263,24 +293,27 @@ try { ```rust use nemo_relay::observability::otel::{OpenTelemetryConfig, OpenTelemetrySubscriber}; -let config = OpenTelemetryConfig::http_binary("agent-service") - .with_endpoint("http://localhost:4318/v1/traces") - .with_resource_attribute("deployment.environment", "dev"); -let subscriber = OpenTelemetrySubscriber::new(config)?; -subscriber.register("otel-exporter")?; +fn main() -> Result<(), Box> { + let config = OpenTelemetryConfig::http_binary("agent-service") + .with_endpoint("http://localhost:4318/v1/traces") + .with_resource_attribute("deployment.environment", "dev"); + let subscriber = OpenTelemetrySubscriber::new(config)?; + subscriber.register("otel-exporter")?; -// Run instrumented application work here. + // Run instrumented application work here. -subscriber.force_flush()?; -let _ = subscriber.deregister("otel-exporter")?; -subscriber.shutdown()?; + subscriber.force_flush()?; + let _ = subscriber.deregister("otel-exporter")?; + subscriber.shutdown()?; + Ok(()) +} ``` -## Common Validation Failures +## Common Configuration and Runtime Issues - `transport` is not `http_binary` or `grpc`. - Headers or resource attributes are not string-to-string maps. diff --git a/docs/pii-redaction-plugin/about.mdx b/docs/configure-plugins/pii-redaction/about.mdx similarity index 83% rename from docs/pii-redaction-plugin/about.mdx rename to docs/configure-plugins/pii-redaction/about.mdx index 2d0231dac..73711e4e8 100644 --- a/docs/pii-redaction-plugin/about.mdx +++ b/docs/configure-plugins/pii-redaction/about.mdx @@ -1,7 +1,7 @@ --- title: "PII Redaction Plugin" sidebar-title: "About" -description: "" +description: "Configure built-in PII redaction for emitted NeMo Relay observability payloads." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -14,7 +14,12 @@ plugin system. The built-in plugin component uses `kind = "pii_redaction"` and is available as a first-party NeMo Relay plugin. -The plugin is designed around backend modes: +The CLI and primary language bindings register this component. Direct Rust +applications must call +`nemo_relay_pii_redaction::component::register_pii_redaction_component()` +before they validate or initialize a `pii_redaction` component. + +The plugin supports these backend modes: - `builtin` - Uses a native Rust backend for deterministic payload sanitization. @@ -94,10 +99,10 @@ That means: - The real provider request and response values remain unchanged. - Subscribers and exporters receive sanitized payloads after the plugin runs. -For managed LLM request payloads, codec decode and re-encode can canonicalize -the emitted provider-shaped start event. For example, an OpenAI Responses -request may be recorded in the codec's canonical `input` array form instead of -the original shorthand request shape. +For managed LLM requests, codec decode and re-encode can canonicalize the +emitted provider-shaped start event. For example, Relay can record an OpenAI +Responses request in the codec's canonical `input` array form rather than the +original shorthand form. ## Current Boundaries @@ -114,6 +119,6 @@ In particular: ## Pages -- [PII Redaction Configuration](/pii-redaction-plugin/configuration) +- [PII Redaction Configuration](/configure-plugins/pii-redaction/configuration) documents the built-in component shape, action semantics, supported codecs, and example configs. diff --git a/docs/pii-redaction-plugin/configuration.mdx b/docs/configure-plugins/pii-redaction/configuration.mdx similarity index 86% rename from docs/pii-redaction-plugin/configuration.mdx rename to docs/configure-plugins/pii-redaction/configuration.mdx index 8beb6861d..1d2c16af6 100644 --- a/docs/pii-redaction-plugin/configuration.mdx +++ b/docs/configure-plugins/pii-redaction/configuration.mdx @@ -1,7 +1,7 @@ --- title: "PII Redaction Configuration" sidebar-title: "Configuration" -description: "" +description: "Configure PII redaction actions, detectors, targets, and backend modes." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -10,9 +10,14 @@ SPDX-License-Identifier: Apache-2.0 */} Use this page when you want to configure the built-in PII redaction plugin component. The component kind is `pii_redaction`. +The CLI and primary language bindings register this component. Direct Rust +applications must call +`nemo_relay_pii_redaction::component::register_pii_redaction_component()` +before they validate or initialize a `pii_redaction` component. + For plugin file discovery, precedence, merge behavior, editor controls, and gateway conflict rules, refer to -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). NeMo Relay plugin configuration uses the generic plugin document shape, so @@ -28,9 +33,9 @@ This plugin uses the same sanitize-guardrail middleware family documented in The difference between raw middleware and `pii_redaction` is the layer of abstraction: -- Raw middleware asks you to register sanitize callbacks directly in code +- Raw middleware asks you to register sanitize callbacks directly in code. - `pii_redaction` gives you a first-party, config-driven privacy contract on - top of those same runtime hooks + top of those same runtime hooks. Choose `pii_redaction` when you want a reusable built-in policy surface. Choose raw middleware when you need bespoke callback logic that does not fit @@ -75,6 +80,8 @@ At least one managed redaction surface must be enabled. ## Backend Support +The following table compares the available PII redaction backends: + | Area | `builtin` | `local_model` | |---|---|---| | Built-in component kind and config validation | Supported | Supported | @@ -86,7 +93,7 @@ At least one managed redaction surface must be enabled. | Codec support | `openai_chat`, `openai_responses`, `anthropic_messages` | Runtime-specific future implementation | | Runtime availability | Any runtime that includes the `nemo-relay-pii-redaction` plugin crate | Runtimes that install a local backend provider | -## Built-In Mode +## Built-in Mode Use `builtin` mode when NeMo Relay should sanitize emitted observability payloads with a deterministic first-party backend. @@ -109,7 +116,7 @@ To use `mode = "builtin"`: You can write this config directly in `plugins.toml`, or create and edit it through the CLI with `nemo-relay plugins edit`. For plugin file discovery, precedence, merge behavior, and editor controls, refer to -[Plugin Configuration Files](/build-plugins/plugin-configuration-files). +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ```toml version = 1 @@ -170,10 +177,10 @@ The editor preserves unknown fields when it rewrites an existing `pii_redaction` component, so future or runtime-specific settings are not discarded by the interactive edit flow. -If you find yourself needing callback code instead of editor/config fields, it -is a sign that raw middleware may be the better fit for that specific policy. +If your policy needs callback code instead of editor or configuration fields, +use raw middleware. -## Built-In Settings +## Built-in Settings The `builtin` section contains: @@ -196,9 +203,9 @@ The `builtin` section contains: When a target matches: -- object fields are removed -- array elements become `null` -- targeted scalar or root values become `null` +- Object fields are removed. +- Array elements become `null`. +- Targeted scalar or root values become `null`. ### `regex_replace` @@ -275,47 +282,39 @@ The current implementation also preserves provider-shaped response-path compatibility for the supported codecs, but normalized LLM paths are the recommended contract for new configuration. -## Choosing Between This Plugin and Middleware - -Use this plugin when: - -- The privacy behavior should be reusable across applications -- Config-driven enablement matters more than hand-written callbacks -- You want built-in detectors and action semantics -- You want a documented first-party NeMo Relay privacy surface - -Use raw middleware when: +## Decision Rule -- The policy depends on application-specific runtime state -- The sanitization logic is too custom for the plugin contract -- You need to prototype or experiment before standardizing behavior - -The runtime effect is still sanitize-guardrail middleware in both cases. The -plugin simply gives you a standardized policy layer on top. +Use `pii_redaction` for reusable, config-driven privacy behavior. Use raw +middleware when the policy depends on application-specific runtime state or +custom callback logic. Refer to [Plugin Versus Middleware](#plugin-versus-middleware) +for the full comparison. ## Detector Presets -The built-in detector presets are grouped into three deterministic families. +The built-in detector presets are grouped into three deterministic families: + +**Common PII** -Common PII: - `email` - `phone` - `ip_address` - `ipv6` - `url` -Structured secrets: +**Structured Secrets** + - `api_key` - `uuid` - `bearer_token` - `jwt` - `credit_card` - `bearer_token` is heuristic rather than vendor-specific. It can still match - benign bearer-style values, so prefer a narrower detector when you know the - credential family. +`bearer_token` is heuristic rather than vendor-specific. It can still match +benign bearer-style values, so prefer a narrower detector when you know the +credential family. + +**Cloud Credentials** -Cloud credentials: - `aws_access_key_id` - `aws_secret_access_key` - `gcp_api_key` @@ -332,10 +331,10 @@ The built-in plugin uses sanitize guardrails. That means: -- the real provider response value is unchanged -- the emitted NeMo Relay start or end event payload is sanitized -- `annotated_response` is populated from the sanitized end-event payload when a - response codec is provided +- The real provider response value remains unchanged. +- The plugin sanitizes emitted NeMo Relay start and end event payloads. +- The plugin populates `annotated_response` from the sanitized end-event payload + when a response codec is available. ## Local Model Mode @@ -343,16 +342,16 @@ That means: ### Current Status -Currently: +The current local-model status is: -- the plugin contract accepts `mode = "local_model"` -- the `local` section currently supports: +- The plugin contract accepts `mode = "local_model"`. +- The `local` section supports: - `backend` - `model_id` - `detector_profile` - `allow_network` - `max_latency_ms` -- actual behavior depends on a runtime-installed local backend provider +- Actual behavior depends on a runtime-installed local backend provider. Without a provider, runtimes report the local backend as unavailable during plugin initialization. diff --git a/docs/build-plugins/plugin-configuration-files.mdx b/docs/configure-plugins/plugin-configuration-files.mdx similarity index 60% rename from docs/build-plugins/plugin-configuration-files.mdx rename to docs/configure-plugins/plugin-configuration-files.mdx index c1a206932..2d5d7e81c 100644 --- a/docs/build-plugins/plugin-configuration-files.mdx +++ b/docs/configure-plugins/plugin-configuration-files.mdx @@ -1,20 +1,22 @@ --- title: "Plugin Configuration Files" -description: "" +description: "Configure plugins.toml discovery, layering, validation, and interactive editing." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -Use `plugins.toml` when the `nemo-relay` CLI gateway should activate plugins at -startup. The file contains the same generic plugin configuration document used -by the Rust, Python, and Node.js plugin APIs, but encoded as TOML at the file -root. +Use `plugins.toml` when Relay should activate plugins from file configuration. +The runtime discovers and layers this generic plugin document for direct Rust, +Python, and Node.js integrations as well as for the `nemo-relay` CLI gateway. +The file encodes the same document that binding APIs accept, but uses TOML at +the file root. -This page documents file discovery, precedence, merge behavior, editor behavior, -and conflict rules for the CLI gateway. Component-specific fields are documented -in the guide for each plugin component. +This guide documents runtime file discovery, precedence, merge behavior, and +conflict rules. It also identifies gateway-only editor, explicit-config, and +discoverable-plugin workflows. Each component guide documents its +component-specific fields. NeMo Relay plugin configuration keys use `snake_case` regardless of language or @@ -26,7 +28,7 @@ generic plugin document and component-local `config` objects use canonical ## Shortest Path Example -Use this minimal `plugins.toml` when you want the CLI gateway to start with one +Use this minimal `plugins.toml` when you want Relay to start with one plugin-managed observability exporter and no extra layering: ```toml @@ -46,9 +48,9 @@ filename = "events.jsonl" mode = "append" ``` -For the most deterministic verification path, keep `plugins.toml` in the same -directory as the gateway `config.toml` file for this run, then launch the -wrapper with that explicit config path. For example: +For the most deterministic CLI-gateway verification path, keep `plugins.toml` +in the same directory as the gateway `config.toml` file for this run, then +launch the wrapper with that explicit config path. For example: ```text path/to/ @@ -56,6 +58,8 @@ path/to/ plugins.toml ``` +Run the gateway with the following command: + ```bash nemo-relay --config path/to/config.toml run -- codex ``` @@ -115,34 +119,69 @@ The top-level fields are: | `components` | `[]` | Ordered plugin components to validate and activate. | | `policy` | warn unknown components and fields, error on unsupported values | Global validation policy. | -Each component has: +Each built-in component has: | Field | Default | Notes | |---|---|---| | `kind` | Required | Registered plugin kind, such as `observability` or `adaptive`. | -| `enabled` | `true` | Disabled components are validated but not initialized. | +| `enabled` | `true` | Relay validates disabled components but does not initialize them. | | `config` | `{}` | Component-local configuration object. The shape depends on `kind`. | -The gateway reads only files named `plugins.toml`. +## Gateway Discoverable Plugin Records + +Use `[[plugins.dynamic]]` only for a gateway-managed manifest-backed native or +worker plugin. These records remain separate from `[[components]]`. During +gateway activation, Relay loads each enabled dynamic adapter, synthesizes its +internal component, and validates the component configuration. + +The following record configures a dynamic plugin: + +```toml +[[plugins.dynamic]] +manifest = "./plugins/acme/relay-plugin.toml" + +[plugins.dynamic.config] +mode = "audit" +``` + + +For a Python worker, run `nemo-relay plugins add ` instead of adding +this record manually. The command creates and records the Relay-managed Python +environment that the worker requires at startup. + + +`manifest` is required and resolves relative to this file. `config` is optional. +Run `nemo-relay plugins validate ` to validate it against the +manifest’s optional static JSON Schema before you enable or run the plugin. Use +`nemo-relay plugins add`, `validate`, `inspect`, and `enable` to manage the +dynamic-plugin lifecycle. Refer to [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) +for manifest, trust, and policy requirements. -## Discovery +The runtime reads only files named `plugins.toml` during default discovery. -The gateway resolves plugin configuration from `plugins.toml` files and an -optional code-driven layer. +## Runtime Discovery + +The runtime resolves plugin configuration from `plugins.toml` files and an +optional code-driven layer. Direct Rust, Python, and Node.js calls to +their plugin-initialization APIs use this same discovery and layering behavior. File configuration comes from `plugins.toml`: | Source | Use case | |---|---| -| `plugins.toml` | Normal operator- and project-managed gateway plugin configuration. | +| `plugins.toml` | Normal operator- and project-managed runtime plugin configuration. | + +The runtime does not read plugin configuration from `config.toml`. -Plugin configuration is not read from `config.toml`. +### Gateway Explicit Config -When `--config path/to/config.toml` is supplied, plugin file discovery is scoped -to `path/to/plugins.toml`. Implicit system, project, and user plugin files are -not loaded for that run. +When the CLI gateway receives `--config path/to/config.toml`, it scopes plugin +file discovery to `path/to/plugins.toml`. It does not load implicit system, +project, or user plugin files for that run. -When no explicit `--config` path is supplied, the gateway checks these +### Default Discovery Locations + +When no gateway `--config` path overrides discovery, the runtime checks these `plugins.toml` locations from lowest to highest precedence: 1. System: `/etc/nemo-relay/plugins.toml` @@ -151,12 +190,14 @@ When no explicit `--config` path is supplied, the gateway checks these 3. User: `$XDG_CONFIG_HOME/nemo-relay/plugins.toml`, or `~/.config/nemo-relay/plugins.toml` when `XDG_CONFIG_HOME` is not set -Missing files are skipped. If no plugin config source exists, the gateway starts -without process-level plugin activation. +The runtime skips missing files. If no plugin config source exists, +initialization continues without process-level plugin activation. -## Editing Files +## Gateway Editing Files -Use the interactive editor for Observability and Adaptive plugin configuration: +Use the interactive editor for Observability, Adaptive, NeMo Guardrails, and +PII Redaction configuration. The editor also updates the configuration of +manifest-backed dynamic plugins that `plugins add` has registered: ```bash nemo-relay plugins edit @@ -207,7 +248,7 @@ The editor menus support these controls: Text and JSON value prompts use normal line editing. Use the surrounding field menu to reset, clear, preview, or save. -## Precedence And Merge Behavior +## Precedence and Merge Behavior When more than one `plugins.toml` file is discovered, later files have higher precedence. User config overrides project config, and project config overrides @@ -226,6 +267,8 @@ output_directory = "/var/log/nemo-relay" mode = "append" ``` +The following user configuration overrides only the exporter mode: + ```toml # user plugins.toml [[components]] @@ -239,10 +282,10 @@ The effective Agent Trajectory Observability Format (ATOF) config keeps `enabled` and `output_directory` from the system file and uses `mode = "overwrite"` from the user file. -The top-level `components` array is special. Components are matched by `kind` +The top-level `components` array is special. Relay matches components by `kind` across files. A higher-precedence component with the same `kind` merges into the -lower-precedence component. A component with a different `kind` is added to the -effective configuration. +lower-precedence component. Relay adds a component with a different `kind` to +the effective configuration. Most components follow the general merge rules above. The built-in `pricing` component has one additional merge rule: when both lower and @@ -255,8 +298,8 @@ Declare each `kind` at most once inside one `plugins.toml` file. Duplicate component kinds in the same file fail before merge. Duplicate singleton components that reach plugin validation also fail validation. -Arrays inside component config are replaced by the higher-precedence value. -Tables inside component config merge recursively. +Higher-precedence values replace arrays inside component config. Higher-precedence +tables merge recursively inside component config. ## Configuration Layering @@ -268,7 +311,7 @@ follows: (system → project → user), using the [Precedence And Merge Behavior](#precedence-and-merge-behavior) rules above. 2. Layer the config object you pass to `initialize` over that merged base. Any setting it specifies overrides the file value, and the result is the - effective config that gets validated and activated. + effective config that Relay validates and activates. Files and code differ only in how they treat a setting you **omit**: @@ -277,19 +320,21 @@ Files and code differ only in how they treat a setting you **omit**: | `version`, `policy`, or the `enabled` flag of a component you declare | Inherited from a lower-precedence file | **Always taken from code** — its default if you did not set it | | A whole component kind, or a key inside a component's `config` | Inherited from a lower-precedence file | Inherited from the file layer | -In other words, the layers beneath a file fill its gaps. Code's typed fields -are never gaps because they always carry a value, so they always take -precedence. Only the open-ended parts of your code config (which components you -include and the keys within each component's `config`) merge with the files. +Lower-precedence files fill fields that higher-precedence files omit. Typed code +fields always have values, so they override file values. Only component +selection and keys inside component `config` merge with files. Without filesystem access, no files are read, so the base is empty and only your `initialize` config applies. -## Explicit Defaults And Overrides +## Explicit Defaults and Overrides The editor writes explicit defaults for edited Observability and Adaptive -sections. This is intentional. In a layered config model, omitting a field means -"inherit a lower precedence value"; it does not mean "delete that value." +sections. It writes NeMo Guardrails, PII Redaction, and dynamic-plugin fields +only when readers configure them. In a layered config model, omitting a field +means "inherit a lower precedence value"; it does not mean "delete that value." +Use the dedicated `nemo-relay model-pricing` commands to manage model-pricing +catalog sources. For example, this user file disables ATOF even if a project file enables it: @@ -302,7 +347,7 @@ enabled = false mode = "append" ``` -The merged config may still contain inherited ATOF sibling fields, such as +The merged config can still contain inherited ATOF sibling fields, such as `output_directory`, but the runtime ignores the section because `enabled = false`. @@ -329,13 +374,20 @@ Common validation failures include: - Enabled components whose build-time features are unavailable. - Component-specific semantic failures, such as an Agent Trajectory Interchange Format (ATIF) filename template that does not contain `{session_id}`. - -Use `nemo-relay doctor` to inspect the resolved gateway configuration and plugin -diagnostics. For Observability, doctor also reports enabled exporter sections, -checks writable file exporter directories, probes configured ATOF streaming -endpoints, and checks reachable OTLP endpoints when those settings are present. -For model pricing, doctor validates enabled file and inline sources and fails -when a source is unreadable or the catalog schema is invalid. +- Dynamic manifests that have an incompatible Relay version, unsupported + capability, invalid load contract, or failed trust evidence. Run + `nemo-relay plugins validate ` to check optional static schema + validation for dynamic component config. + +Use `nemo-relay doctor` to inspect the resolved gateway configuration and +plugin diagnostics. For Observability, doctor also reports enabled exporter +sections, checks writable file exporter directories, probes configured ATOF +streaming endpoints, and checks reachable OTLP endpoints when those settings +are present. For model pricing, doctor validates enabled file and inline +sources and fails when a source is unreadable or the catalog schema is invalid. +For dynamic-plugin validation, lifecycle state, and trust diagnostics, use +`nemo-relay plugins validate`, `inspect`, and `list`; doctor reports only the +resolved manifest references and host configuration status. ## Relationship to `config.toml` @@ -354,7 +406,10 @@ Observability exporters through `plugins.toml`. Use the component guides for field-level configuration: -- [Observability Configuration](/observability-plugin/configuration) -- [Adaptive Configuration](/adaptive-plugin/configuration) -- [Adaptive Cache Governor (ACG)](/adaptive-plugin/acg) -- [Adaptive Hints](/adaptive-plugin/adaptive-hints) +- [Observability Configuration](/configure-plugins/observability/configuration) +- [Adaptive Configuration](/configure-plugins/adaptive/configuration) +- [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg) +- [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) +- [NeMo Guardrails Configuration](/configure-plugins/nemo-guardrails/configuration) +- [PII Redaction Configuration](/configure-plugins/pii-redaction/configuration) +- [Model Pricing](/configure-plugins/model-pricing) diff --git a/docs/contribute/runtime-contract-docs.mdx b/docs/contribute/runtime-contract-docs.mdx index 7fe991da8..ee1a0022c 100644 --- a/docs/contribute/runtime-contract-docs.mdx +++ b/docs/contribute/runtime-contract-docs.mdx @@ -1,6 +1,6 @@ --- title: "Runtime Contract Documentation" -description: "" +description: "Keep NeMo Relay runtime-contract documentation consistent across bindings, integrations, and plugins." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -31,11 +31,11 @@ These pages define or route the shared runtime model: - `docs/getting-started/installation.mdx` - `docs/getting-started/quick-start/index.mdx` - `docs/reference/api/index.mdx` -- `docs/observability-plugin/about.mdx` -- `docs/observability-plugin/atof.mdx` -- `docs/observability-plugin/atif.mdx` -- `docs/observability-plugin/opentelemetry.mdx` -- `docs/observability-plugin/openinference.mdx` +- `docs/configure-plugins/observability/about.mdx` +- `docs/configure-plugins/observability/atof.mdx` +- `docs/configure-plugins/observability/atif.mdx` +- `docs/configure-plugins/observability/opentelemetry.mdx` +- `docs/configure-plugins/observability/openinference.mdx` - `docs/resources/glossary.mdx` Support-sensitive summaries should also stay aligned with: @@ -72,7 +72,7 @@ or cost policy. Codecs translate typed application values or provider-native payloads into stable runtime shapes. They do not decide ownership, middleware ordering, -whether execution may continue, or which exporter is active. +whether execution can continue, or which exporter is active. ### Managed Versus Explicit Integration diff --git a/docs/getting-started/agent-runtime-primer.mdx b/docs/getting-started/agent-runtime-primer.mdx index 33ee3dd35..36dacbb70 100644 --- a/docs/getting-started/agent-runtime-primer.mdx +++ b/docs/getting-started/agent-runtime-primer.mdx @@ -1,6 +1,6 @@ --- title: "Agent Runtime Primer" -description: "" +description: "Learn the NeMo Relay runtime model for scopes, middleware, events, and integration boundaries." position: 1 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -115,5 +115,5 @@ configuration, and validation steps for that path. - **Reusable runtime behavior across services or teams:** Runtime plugin configuration owns reusable middleware, subscribers, exporters, model pricing, policy, or adaptive behavior. Start with [Build Plugins](/build-plugins/about), - use [Observability](/observability-plugin/about) for export setup, and use - [Adaptive](/adaptive-plugin/about) after baseline instrumentation is working. + use [Observability](/configure-plugins/observability/about) for export setup, and use + [Adaptive](/configure-plugins/adaptive/about) after baseline instrumentation is working. diff --git a/docs/getting-started/configuration.mdx b/docs/getting-started/configuration.mdx index 8455b303d..a77701e5d 100644 --- a/docs/getting-started/configuration.mdx +++ b/docs/getting-started/configuration.mdx @@ -1,7 +1,7 @@ --- title: "Configuration" sidebar-title: "Configuration / Setup" -description: "" +description: "Configure NeMo Relay runtime behavior, plugins, observability, and adaptive features." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -29,13 +29,13 @@ Plugins use a structured plugin configuration with: - One or more component definitions - Optional component policy -Start with [Define a Plugin](/build-plugins/basic-guide) when you need reusable middleware, subscribers, or adaptive behavior. +Start with [Language Binding Plugins](/build-plugins/language-binding/about) when you need reusable middleware, subscribers, or adaptive behavior. -Use [NeMo Guardrails Configuration](/nemo-guardrails-plugin/configuration) when +Use [NeMo Guardrails Configuration](/configure-plugins/nemo-guardrails/configuration) when you want the built-in first-party `nemo_guardrails` component. -The `nemo-relay` CLI gateway reads plugin files named `plugins.toml`. See -[Plugin Configuration Files](/build-plugins/plugin-configuration-files) +The `nemo-relay` CLI gateway reads plugin files named `plugins.toml`. Refer to +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files) for file locations, precedence, merge behavior, editor controls, and validation rules. @@ -45,9 +45,9 @@ Agent Trajectory Observability Format (ATOF) exporters, Agent Trajectory Interchange Format (ATIF) exporters, OpenTelemetry subscribers, and OpenInference subscribers can be configured directly through binding-native config objects. Use the built-in `observability` plugin when you want one -plugin component to own standard exporter setup and teardown. See -[Observability Configuration](/observability-plugin/configuration) -and [Observability](/observability-plugin/about) +plugin component to own standard exporter setup and teardown. Refer to +[Observability Configuration](/configure-plugins/observability/configuration) +and [Observability](/configure-plugins/observability/about) for the supported export paths. NeMo Relay does not require application-level environment variables for normal @@ -61,4 +61,4 @@ deployment manifests. ## Adaptive Setup -Adaptive tuning is enabled through the adaptive plugin component and binding helper APIs. See [Adaptive Configuration](/adaptive-plugin/configuration). +Adaptive tuning is enabled through the adaptive plugin component and binding helper APIs. Refer to [Adaptive Configuration](/configure-plugins/adaptive/configuration). diff --git a/docs/getting-started/installation.mdx b/docs/getting-started/installation.mdx index fd3254568..a8bf387ab 100644 --- a/docs/getting-started/installation.mdx +++ b/docs/getting-started/installation.mdx @@ -69,7 +69,7 @@ nemo-relay install claude-code nemo-relay install codex ``` -See [Plugin Installation](/nemo-relay-cli/plugin-installation) for the host +Refer to [Plugin Installation](/nemo-relay-cli/plugin-installation) for the host plugin workflow. diff --git a/docs/getting-started/prerequisites.mdx b/docs/getting-started/prerequisites.mdx index 0b160592f..770096962 100644 --- a/docs/getting-started/prerequisites.mdx +++ b/docs/getting-started/prerequisites.mdx @@ -14,8 +14,8 @@ Install the tooling for the binding you plan to use. | Rust | 1.86 or newer | Rust builds, local workspace builds, and the Rust core runtime | | Python | 3.11 or newer | Python bindings, Python tests, and docs tooling | | Node.js | 24 or newer | Node.js bindings and generated Node.js API docs | -| `uv` | see [Development Setup](/contribute/development-setup) | Python environments, docs builds, and repository setup | -| `just` | see [Development Setup](/contribute/development-setup) | Repository development, test, build, and docs task aliases | +| `uv` | Refer to [Development Setup](/contribute/development-setup). | Python environments, docs builds, and repository setup | +| `just` | Refer to [Development Setup](/contribute/development-setup). | Repository development, test, build, and docs task aliases | The primary documentation track covers Rust, Python, and Node.js. Go and the raw FFI surface are experimental and source-first. diff --git a/docs/getting-started/quick-start/index.mdx b/docs/getting-started/quick-start/index.mdx index bfa1204a0..f1152f1e4 100644 --- a/docs/getting-started/quick-start/index.mdx +++ b/docs/getting-started/quick-start/index.mdx @@ -1,18 +1,18 @@ --- title: "Quick Start" -description: "" +description: "Choose a NeMo Relay quick start for CLI, application, framework, or plugin-managed integration." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -Choose the Quick Start path from the following for the boundary you want Relay -to observe or control. +Choose a Quick Start path based on the boundary that you want Relay to observe +or control: -- Use the CLI path when a local coding-agent harness owns the invocation -- Use a binding quick start when your application owns the tool or LLM call +- Use the CLI path when a local coding-agent harness owns the invocation. +- Use a binding quick start when your application owns the tool or LLM call. - Use a supported integration guide when a framework or agent harness owns the - boundary + boundary. If you still need the shared runtime vocabulary for scopes, middleware, events, or plugins, refer to the [Agent Runtime Primer](/getting-started/agent-runtime-primer) @@ -29,11 +29,11 @@ not yet know which guide owns the working path. | Persistent host-plugin installs | You want the maintained install path for Codex or Claude Code instead of the transparent wrapper | [Plugin Installation](/nemo-relay-cli/plugin-installation) | `nemo-relay doctor --plugin ` confirms the installed host plugin is registered and ready. | | Direct Python or Node.js application APIs | Your application owns the tool or LLM callback | [Python Quick Start](/getting-started/quick-start/python) or [Node.js Quick Start](/getting-started/quick-start/nodejs) | The sample prints event lines plus tool and LLM results. | | Direct Rust application APIs | Your Rust application owns the tool or LLM callback | [Rust Quick Start](/getting-started/quick-start/rust) | The sample prints scope, tool, and LLM lifecycle output plus the `initialized` mark event. | -| Plugin-managed runtime setup | You need process-level exporter or plugin behavior from `plugins.toml` | [Plugin Configuration Files](/build-plugins/plugin-configuration-files) | The selected plugin path activates and writes the expected output or behavior. | +| Plugin-managed runtime setup | You need process-level exporter or plugin behavior from `plugins.toml` | [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) | The selected plugin path activates and writes the expected output or behavior. | | Managed middleware | You want policy, redaction, routing, or execution wrapping around managed calls | [Add Middleware](/instrument-applications/advanced-guide) | One allowed request succeeds, one rejected request stops before execution, and observed payloads match the policy. | | Framework integrations | A framework such as LangChain, LangGraph, or Deep Agents owns callbacks or scheduling | [Supported Integrations](/supported-integrations/about) | The integration guide's verify step confirms the expected framework-owned output. | | OpenClaw plugin path | OpenClaw owns plugin setup and Relay observes the OpenClaw-managed boundary | [OpenClaw](/supported-integrations/openclaw-plugin) | The OpenClaw guide's verify step confirms plugin setup and runtime output. | -| Manual or CI workflows | You need explicit config files, deterministic commands, or non-interactive automation | [CLI Basic Usage](/nemo-relay-cli/basic-usage) and [Plugin Configuration Files](/build-plugins/plugin-configuration-files) | The explicit command uses the intended config files without interactive setup. | +| Manual or CI workflows | You need explicit config files, deterministic commands, or non-interactive automation | [CLI Basic Usage](/nemo-relay-cli/basic-usage) and [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) | The explicit command uses the intended config files without interactive setup. | ## Local Coding-Agent Runs diff --git a/docs/index.yml b/docs/index.yml index bcdcaba31..fe797eaa2 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -22,30 +22,58 @@ navigation: title: "Instrument Applications" slug: instrument-applications title-source: frontmatter - - folder: ./observability-plugin - title: "Observability Plugin" - slug: observability-plugin - title-source: frontmatter - - folder: ./adaptive-plugin - title: "Adaptive Plugin" - slug: adaptive-plugin - title-source: frontmatter - - folder: ./nemo-guardrails-plugin - title: "NeMo Guardrails Plugin" - slug: nemo-guardrails-plugin - title-source: frontmatter - - folder: ./pii-redaction-plugin - title: "PII Redaction Plugin" - slug: pii-redaction-plugin - title-source: frontmatter + - section: "Configure Plugins" + slug: configure-plugins + contents: + - page: "About" + path: ./configure-plugins/about.mdx + slug: about + - folder: ./configure-plugins/observability + title: "Observability" + title-source: frontmatter + - folder: ./configure-plugins/adaptive + title: "Adaptive" + title-source: frontmatter + - folder: ./configure-plugins/nemo-guardrails + title: "NeMo Guardrails" + title-source: frontmatter + - folder: ./configure-plugins/pii-redaction + title: "PII Redaction" + title-source: frontmatter + - page: "Model Pricing" + path: ./configure-plugins/model-pricing.mdx + slug: model-pricing + - page: "Configure Discoverable Plugins" + path: ./configure-plugins/discoverable-plugins.mdx + slug: discoverable-plugins + - page: "Plugin Configuration Files" + path: ./configure-plugins/plugin-configuration-files.mdx + slug: plugin-configuration-files - folder: ./integrate-into-frameworks title: "Integrate into Frameworks" slug: integrate-into-frameworks title-source: frontmatter - - folder: ./build-plugins - title: "Build Plugins" + - section: "Build Plugins" slug: build-plugins - title-source: frontmatter + contents: + - page: "About" + path: ./build-plugins/about.mdx + slug: about + - folder: ./build-plugins/language-binding + title: "Language Binding" + title-source: frontmatter + - section: "Dynamic Plugins" + slug: dynamic-plugins + contents: + - page: "Discoverable Plugins" + path: ./build-plugins/dynamic-plugins/about.mdx + slug: about + - folder: ./build-plugins/dynamic-plugins/native-dynamic + title: "Native Dynamic" + title-source: frontmatter + - folder: ./build-plugins/dynamic-plugins/grpc-worker + title: "gRPC Worker" + title-source: frontmatter - folder: ./contribute title: "Contribute" slug: contribute diff --git a/docs/instrument-applications/instrument-llm-call.mdx b/docs/instrument-applications/instrument-llm-call.mdx index f8159c684..0d75aadcf 100644 --- a/docs/instrument-applications/instrument-llm-call.mdx +++ b/docs/instrument-applications/instrument-llm-call.mdx @@ -1,6 +1,6 @@ --- title: "Instrument an LLM Call" -description: "" +description: "Instrument managed LLM calls with NeMo Relay scopes, middleware, events, and subscribers." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -278,4 +278,4 @@ Use these links to continue from this workflow into the next related task. - Use [Provider Codecs](/integrate-into-frameworks/provider-codecs) when request intercepts need normalized LLM request data or downstream consumers need normalized response annotations. -- Export events with [Observability](/observability-plugin/about). +- Export events with [Observability](/configure-plugins/observability/about). diff --git a/docs/instrument-applications/instrument-tool-call.mdx b/docs/instrument-applications/instrument-tool-call.mdx index 1fe82d046..e9dbad4f8 100644 --- a/docs/instrument-applications/instrument-tool-call.mdx +++ b/docs/instrument-applications/instrument-tool-call.mdx @@ -1,6 +1,6 @@ --- title: "Instrument a Tool Call" -description: "" +description: "Instrument managed tool calls with NeMo Relay scopes, middleware, events, and subscribers." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -228,5 +228,5 @@ Use these links to continue from this workflow into the next related task. - Add model-provider instrumentation with [Instrument an LLM Call](/instrument-applications/instrument-llm-call). - Add policy or transformation with [Add Middleware](/instrument-applications/advanced-guide). -- Export events with [Observability](/observability-plugin/about). +- Export events with [Observability](/configure-plugins/observability/about). - Use [Code Examples](/instrument-applications/code-examples) for manual lifecycle, streaming, scope, and partial middleware API examples. diff --git a/docs/integrate-into-frameworks/provider-response-codecs.mdx b/docs/integrate-into-frameworks/provider-response-codecs.mdx index ec78a0068..5111e201e 100644 --- a/docs/integrate-into-frameworks/provider-response-codecs.mdx +++ b/docs/integrate-into-frameworks/provider-response-codecs.mdx @@ -1,6 +1,6 @@ --- title: "Provider Response Codecs and Model Pricing" -description: "" +description: "Normalize LLM response data with provider codecs and configure model pricing estimates." position: 8 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -18,7 +18,7 @@ Response codecs are observability-only: - They do not rewrite the value returned to the application. - They do not run response middleware. - They attach normalized response data to lifecycle events for subscribers and exporters. -- Decode failures are non-fatal; the LLM call still returns the provider response and the end event is emitted without an annotation. +- Decode failures are non-fatal. The LLM call still returns the provider response, and Relay emits the end event without an annotation. ## Before You Start @@ -40,7 +40,7 @@ Response codecs normalize provider output into fields that subscribers can inspe | `message` | Primary assistant message content. | | `tool_calls` | Tool calls requested by the model. | | `finish_reason` | Normalized completion reason, such as `complete`, `length`, `tool_use`, or `content_filter`. | -| `usage` | Token accounting, including cache-read and cache-write counts when available. May also include normalized `cost` when the provider reports cost or Relay can estimate it from known model pricing. | +| `usage` | Token accounting, including cache-read and cache-write counts when available. It can also include normalized `cost` when the provider reports cost or Relay can estimate it from known model pricing. | | `api_specific` | Provider-specific fields that do not fit the common model. | | `extra` | Additional unmodeled response fields. | @@ -69,8 +69,8 @@ Unknown model pricing and missing token data are non-fatal: Relay omits the cost field and still exports token metrics and response annotations. Relay resolves model pricing through an active `PricingResolver` source chain. -Provider- or framework-reported cost remains authoritative; the resolver is -used only when `Usage.cost` is missing. Relay does not ship provider price data +Provider or framework-reported cost remains authoritative. The resolver runs +only when `Usage.cost` is missing. Relay does not ship provider price data by default: estimates require a configured inline, file, or embedding-provided model pricing source. With no configured source, every model is treated as unknown for model pricing. @@ -78,8 +78,8 @@ unknown for model pricing. Model pricing is runtime state, not a CLI-only feature. Any host that initializes Relay plugins can activate the built-in `pricing` component before it runs managed LLM calls. This includes application code, eval harnesses, custom -agents, and framework integrations. The CLI commands below -are a file-management convenience for the local gateway; embedded hosts can pass +agents, and framework integrations. The CLI commands below are a +file-management convenience for the local gateway. Embedded hosts can pass the same component config directly through the plugin APIs. Source precedence is deployment controlled: @@ -117,7 +117,7 @@ Each catalog entry declares: - `provider` and canonical `model_id`. - `aliases` for dated or provider-specific model IDs. - `currency`, defaulting to `USD`. -- `unit`, defaulting to `per_token`. Relay estimates only `per_token` entries in this version; `per_request`, `per_second`, and `gpu_hour` are representable for future source integrations but are not estimated. +- `unit`, defaulting to `per_token`. Relay estimates only `per_token` entries in this version. `per_request`, `per_second`, and `gpu_hour` are representable for future source integrations but are not estimated. - `rates` per one million input, output, cache-read, and cache-write tokens for flat `per_token` entries. - `rate_schedule` for data-driven threshold-based model pricing, such as models whose full-request input/output rates change after a prompt-token threshold. - `prompt_cache.read_accounting`, which tells Relay whether cache-read tokens are already included in prompt tokens. @@ -127,7 +127,7 @@ Relay validates catalogs at startup and rejects duplicate canonical IDs or aliases within the same normalized provider/model key. The same model ID can appear under distinct providers, such as `openai/gpt-4o-mini` and `azure/openai/gpt-4o-mini`. Adding a model should be a catalog/source update -plus tests; it should not require adding another Rust `match` arm. +plus tests. It should not require adding another Rust `match` arm. Use the CLI to validate catalog files and manage file-backed model pricing sources: @@ -139,14 +139,14 @@ nemo-relay model-pricing resolve gpt-4o-mini --provider openai --prompt-tokens 1 ``` `model-pricing init` creates or enables the `pricing` plugin component in the -selected `plugins.toml`. The initialized component has an empty `sources` list; -use `model-pricing add-source` or an inline config edit to provide model pricing data. +selected `plugins.toml`. The initialized component has an empty `sources` list. +Use `model-pricing add-source` or an inline config edit to provide model pricing data. `model-pricing add-source` validates the referenced JSON catalog before updating `plugins.toml`. It creates the `pricing` component if needed and prepends the new file source by default, making it the highest-priority source in that scope. Use `--append` when the file should be a lower-priority fallback. Both commands -default to user config at `$XDG_CONFIG_HOME/nemo-relay/plugins.toml`; pass +default to user config at `$XDG_CONFIG_HOME/nemo-relay/plugins.toml`. Pass `--project` for `.nemo-relay/plugins.toml` or `--global` for `/etc/nemo-relay/plugins.toml`. @@ -154,7 +154,7 @@ default to user config at `$XDG_CONFIG_HOME/nemo-relay/plugins.toml`; pass reports the winning catalog source, matched provider/model, and, when token counts are supplied, the estimated total cost. The source line is one of `file:` or `inline:`, which makes overlapping project/user/fleet -entries debuggable. This is a dry diagnostic command; it does not mutate +entries debuggable. This is a dry diagnostic command. It does not mutate configuration. `nemo-relay doctor` also validates enabled model pricing sources and reports missing, @@ -172,7 +172,7 @@ configured while still allowing generic model pricing to apply to routed names. For threshold-based model pricing, use `rate_schedule.type = "prompt_token_threshold"`. Relay selects exactly one tier from `prompt_tokens` and applies that tier to the -full request; it does not price only the overflow tokens at the higher rate. +full request. It does not price only the overflow tokens at the higher rate. This matches providers that publish "short context" and "long context" prices for the entire request/session. If `prompt_tokens` is missing for a thresholded entry, Relay omits the estimate instead of guessing. @@ -338,7 +338,7 @@ No exporter emits a running cross-call total other than ATIF `final_metrics`. ### Usage Fields -All normalized fields are optional. A provider may omit a field, while a codec can +All normalized fields are optional. A provider can omit a field, while a codec can compute one (such as Anthropic's `total_tokens`) and configured pricing can synthesize `cost`. `Usage` has no catch-all field, so provider usage fields that Relay does not model are dropped. @@ -386,7 +386,7 @@ currencies. ### Exporter Field Mapping -Each exporter projects `usage`/`cost` differently; projections do not change the +Each exporter projects `usage`/`cost` differently. Projections do not change the canonical fields above. | | ATOF | ATIF step / `final_metrics` | OpenInference | OpenTelemetry | @@ -402,15 +402,15 @@ only when it is USD-denominated and otherwise omit it. ATIF derives metrics from codec-normalized usage where available and fills missing supported fields from the raw payload. `metrics.extra` holds only unmapped keys from the raw `usage`/`token_usage` object (for example reasoning token counts, or a raw -`total_tokens`), and only when the step already has a recognized metric; -normalized-only or total-only values are not projected. +`total_tokens`), and only when the step already has a recognized metric. +Normalized-only or total-only values are not projected. ### Stability The `Usage` and `CostEstimate` field names and meanings, and the exporter mappings above, are stable as of ATOF `0.1` (ATIF schema `ATIF-v1.7`, pricing -catalog `version: 1`). Future NeMo Relay releases may add new optional fields to -the serialized JSON/ATOF shapes; renames or removals are breaking changes and are +catalog `version: 1`). Future NeMo Relay releases can add new optional fields to +the serialized JSON/ATOF shapes. Renames or removals are breaking changes and are called out in release notes. @@ -419,7 +419,7 @@ exhaustive, so adding a field or variant is a source-breaking change for Rust consumers. -The following behaviors are intentional in this release but may change later: +The following behaviors are intentional in this release but can change later: - OpenTelemetry emits cost only, not token counts. - ATIF and OpenInference report cost only in USD. @@ -730,4 +730,4 @@ Use these links to continue from this workflow into the next related task. - Use [Provider Codecs](/integrate-into-frameworks/provider-codecs) for request-side provider codecs and full request/response examples. - Use [Wrap LLM Calls](/integrate-into-frameworks/wrap-llm-calls) to add the managed LLM boundary first. -- Use [Observability](/observability-plugin/about) after annotations are visible in local subscribers. +- Use [Observability](/configure-plugins/observability/about) after annotations are visible in local subscribers. diff --git a/docs/integrate-into-frameworks/wrap-tool-calls.mdx b/docs/integrate-into-frameworks/wrap-tool-calls.mdx index df68460d1..5c7561ac8 100644 --- a/docs/integrate-into-frameworks/wrap-tool-calls.mdx +++ b/docs/integrate-into-frameworks/wrap-tool-calls.mdx @@ -124,7 +124,7 @@ async fn run_framework_tool() -> anyhow::Result { Use explicit lifecycle APIs only when the framework owns the real tool invocation internally and exposes only start and finish hooks. In that case, the integration must preserve the returned handle and call the matching end helper on every success and failure path. -Use standalone request-intercept or conditional-execution helpers when the framework needs only partial middleware behavior before it continues down its own invocation path. See [Code Examples](/integrate-into-frameworks/code-examples#fallback-explicit-api-calls) for those fallback surfaces. +Use standalone request-intercept or conditional-execution helpers when the framework needs only partial middleware behavior before it continues down its own invocation path. Refer to [Code Examples](/integrate-into-frameworks/code-examples#fallback-explicit-api-calls) for those fallback surfaces. ## Validate the Tool Wrapper diff --git a/docs/nemo-relay-cli/basic-usage.mdx b/docs/nemo-relay-cli/basic-usage.mdx index 171bf3cd2..060ecad2f 100644 --- a/docs/nemo-relay-cli/basic-usage.mdx +++ b/docs/nemo-relay-cli/basic-usage.mdx @@ -112,7 +112,7 @@ nemo-relay doctor --plugin codex nemo-relay uninstall codex ``` -See [Plugin Installation](/nemo-relay-cli/plugin-installation) for install +Refer to [Plugin Installation](/nemo-relay-cli/plugin-installation) for install directories, host-specific behavior, and Codex lazy sidecar constraints. ## Shared Configuration diff --git a/docs/nemo-relay-cli/claude-code.mdx b/docs/nemo-relay-cli/claude-code.mdx index 9d5e86838..abc80726a 100644 --- a/docs/nemo-relay-cli/claude-code.mdx +++ b/docs/nemo-relay-cli/claude-code.mdx @@ -61,7 +61,7 @@ nemo-relay doctor --plugin claude-code nemo-relay uninstall claude-code ``` -See [Plugin Installation](/nemo-relay-cli/plugin-installation) for install +Refer to [Plugin Installation](/nemo-relay-cli/plugin-installation) for install directories, rollback behavior, and source marketplace notes. ## Shared Config diff --git a/docs/nemo-relay-cli/hermes.mdx b/docs/nemo-relay-cli/hermes.mdx index 8cf3b405a..28e81fb4f 100644 --- a/docs/nemo-relay-cli/hermes.mdx +++ b/docs/nemo-relay-cli/hermes.mdx @@ -57,7 +57,7 @@ Pass Hermes arguments after `--`: nemo-relay hermes -- chat --provider custom ``` -Once NeMo Relay config exists, this shortcut is equivalent to +After NeMo Relay config exists, this shortcut is equivalent to `nemo-relay run --agent hermes`. The wrapper starts a gateway on a dynamic `127.0.0.1` port and exports `NEMO_RELAY_GATEWAY_URL` for the launched process. After initial NeMo Relay setup exists, Hermes hook configuration is diff --git a/docs/nemo-relay-cli/plugin-installation.mdx b/docs/nemo-relay-cli/plugin-installation.mdx index 081dd0a09..920887fee 100644 --- a/docs/nemo-relay-cli/plugin-installation.mdx +++ b/docs/nemo-relay-cli/plugin-installation.mdx @@ -1,25 +1,24 @@ --- title: "Plugin Installation" -description: "" +description: "Install and manage NeMo Relay host plugins for Claude Code and Codex." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -Use the plugin installation when you want Claude Code or Codex to load NeMo Relay -through the host's normal plugin system instead of launching the agent through a -`nemo-relay` wrapper command. +Install a host plugin when you want Claude Code or Codex to load NeMo Relay +through its normal plugin system instead of through a `nemo-relay` wrapper +command. -The installed plugin keeps the same observability model as transparent runs. The -host hooks provide agent, subagent, tool, prompt, compaction, and stop -lifecycle signals, while model-provider routing sends LLM traffic through the -local NeMo Relay gateway. Hooks alone cannot produce complete LLM request and +The installed plugin emits agent, subagent, tool, prompt, compaction, and stop +lifecycle signals. Model-provider routing sends LLM traffic through the local +NeMo Relay gateway. Hooks alone cannot capture complete LLM request and response spans. ## Requirements -To begin, install `nemo-relay` to make sure it is available on `PATH` or `%PATH%`. +Install `nemo-relay` and ensure that it is available on `PATH` or `%PATH%`. The plugin installer does not download a second Relay binary, install a daemon, or require a plugin-local executable. @@ -30,10 +29,13 @@ The selected host CLI must also be available: ## Install Host Plugin -To install one host plugin, enter: +Run the command for the host plugin that you want to install: ```bash nemo-relay install claude-code +``` + +```bash nemo-relay install codex ``` @@ -69,8 +71,9 @@ registers the generated `nemo-relay-plugin` package with the selected host. For Claude Code, install registers the local Claude marketplace, installs `nemo-relay-plugin@nemo-relay-local` at user scope, and enables provider routing -through the local NeMo Relay sidecar. Existing Claude auth and model settings -are preserved unless they must be backed up to add the Relay provider route. +through the local NeMo Relay sidecar. During installation, NeMo Relay preserves +existing Claude authentication and model settings and backs them up only when +it adds the Relay provider route. For Codex, install registers the local Codex marketplace, installs `nemo-relay-plugin@nemo-relay-local`, enables Codex hooks, merges generated hook @@ -84,19 +87,25 @@ installed Codex hook runs, reuses an already healthy sidecar when one exists, and exits after its idle timeout. -A complete first-request capture in Codex depends on Codex firing an installed -hook before the first provider request. If a Codex version calls the provider -before any hook, the first request cannot be guaranteed under hook-only lazy -startup. +The plugin captures the first provider request only when Codex fires an +installed hook before that request. If a Codex version calls the provider +before any hook, the plugin cannot guarantee first-request capture under +hook-only lazy startup. ## Diagnose -Run the plugin doctor after installation: +Run the command for the installed host that you want to diagnose: ```bash nemo-relay doctor --plugin claude-code +``` + +```bash nemo-relay doctor --plugin codex +``` + +```bash nemo-relay doctor --plugin all ``` @@ -120,11 +129,17 @@ host plugin. ## Uninstall -Remove an installed plugin and restore host configuration: +Run the command for the installed host plugin that you want to remove: ```bash nemo-relay uninstall claude-code +``` + +```bash nemo-relay uninstall codex +``` + +```bash nemo-relay uninstall all ``` @@ -142,8 +157,8 @@ validation: - `.agents/plugins/marketplace.json` Those manifests are useful when validating host plugin metadata from a source -checkout. For end-user setup, we recommend using `nemo-relay install ` because it -generates the local marketplace, registers the host plugin, and performs the -required provider and hook setup together. Avoid keeping both a source-installed +checkout. For end-user setup, use `nemo-relay install `. It generates the +local marketplace, registers the host plugin, and performs the required +provider and hook setup. Avoid keeping both a source-installed plugin and a generated install active for the same host because both can forward the same hook payload. diff --git a/docs/reference/llm-request-intercept-outcomes.mdx b/docs/reference/llm-request-intercept-outcomes.mdx index 57ffa0300..f1012553f 100644 --- a/docs/reference/llm-request-intercept-outcomes.mdx +++ b/docs/reference/llm-request-intercept-outcomes.mdx @@ -78,7 +78,7 @@ or object shape: - Python callbacks return `LLMRequestInterceptOutcome`. - Rust callbacks return `LlmRequestInterceptOutcome`. - Go callbacks return `LLMRequestInterceptOutcome`. -- Node.js and WebAssembly callbacks return `{ request, annotated?, pendingMarks? }`. +- Node.js callbacks return `{ request, annotated?, pendingMarks? }`. JavaScript pending-mark DTOs use `categoryProfile`; canonical JSON retains `pending_marks` and `category_profile`. - Public C callbacks return one owned canonical outcome JSON string, and native @@ -108,7 +108,8 @@ codec input, sanitizer input, or start payload. ## Migration This finalizes unpublished native ABI v1 and `grpc-v1` contracts. Rebuild all -development native plugins and workers. Replace tuple results, split C/Go +development native plugins and workers against the same NeMo Relay release that +hosts them. Replace tuple results, split C/Go outputs, metadata envelopes, and parallel mark-aware registrations with the canonical outcome and the existing `register_llm_request_intercept` registration name. @@ -116,3 +117,4 @@ registration name. ## Related Topics - [Tool Execution Intercept Outcomes](/reference/tool-execution-intercept-outcomes) +- [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) diff --git a/docs/reference/tool-execution-intercept-outcomes.mdx b/docs/reference/tool-execution-intercept-outcomes.mdx index 483477a21..ca177fe83 100644 --- a/docs/reference/tool-execution-intercept-outcomes.mdx +++ b/docs/reference/tool-execution-intercept-outcomes.mdx @@ -31,8 +31,8 @@ outcome succeeds. There is no mark-specific registration path. Use the existing global, scope-local, or plugin-context tool execution registration APIs and return the -canonical outcome from every registered callback. Legacy raw intercept returns -are rejected at public and dynamic-plugin boundaries. +canonical outcome from every registered callback. Relay rejects legacy raw +intercept returns at public and dynamic-plugin boundaries. ## Managed Lifecycle @@ -68,10 +68,11 @@ Canonical JSON uses `pending_marks` and `category_profile` across bindings. This finalizes the unpublished tool execution intercept contract. Update every registered tool execution intercept to return the canonical outcome, while leaving the default tool callback and `next(args)` continuation as raw JSON. -Rebuild development native plugins and workers against the current Relay main -branch. +Rebuild development native plugins and workers against the same NeMo Relay +release that hosts them. ## Related Topics - [LLM Request Intercept Outcomes](/reference/llm-request-intercept-outcomes) - [Add Middleware](/instrument-applications/advanced-guide) +- [gRPC Worker Protocol Overview](/build-plugins/dynamic-plugins/grpc-worker/grpc-worker-protocol) diff --git a/docs/resources/support-and-faqs.mdx b/docs/resources/support-and-faqs.mdx index 642134fe5..37cf44214 100644 --- a/docs/resources/support-and-faqs.mdx +++ b/docs/resources/support-and-faqs.mdx @@ -1,6 +1,6 @@ --- title: "Support and FAQs" -description: "" +description: "Find answers about NeMo Relay bindings, middleware, observability, plugins, and troubleshooting." position: 1 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -200,7 +200,7 @@ subscriber payloads consistent. Use manual lifecycle APIs only when the framework owns the invocation internally and you only have start and finish hooks. In that case, preserve the start -handle, emit the matching end event, and handle error paths deliberately. See +handle, emit the matching end event, and handle error paths deliberately. Refer to [Invocation API Selection](/instrument-applications/code-examples#invocation-api-selection) and [Preferred Integration Order](/integrate-into-frameworks/code-examples#preferred-integration-order). @@ -311,7 +311,7 @@ configuration and source code, and emit summarized metadata when full request or response bodies are not needed. Refer to [Middleware](/about-nemo-relay/concepts/middleware#guardrails) and -[Observability](/observability-plugin/about). +[Observability](/configure-plugins/observability/about). ## Observability And Export @@ -325,7 +325,7 @@ operational tracing, trajectory export, or analytics. Refer to [Subscribers](/about-nemo-relay/concepts/subscribers), [Events](/about-nemo-relay/concepts/events), and -[Observability](/observability-plugin/about). +[Observability](/configure-plugins/observability/about). ### Which Exporter Should I Use? @@ -335,10 +335,10 @@ OpenInference when your tracing stack expects OpenInference-style agent and LLM semantics. Use Agent Trajectory Interchange Format (ATIF) when you need trajectory artifacts for analysis, replay, or evaluation workflows. -Refer to [Exporter Selection](/observability-plugin/about#exporter-selection), -[OpenTelemetry](/observability-plugin/opentelemetry), -[OpenInference](/observability-plugin/openinference), and -[Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif). +Refer to [Exporter Selection](/configure-plugins/observability/about#exporter-selection), +[OpenTelemetry](/configure-plugins/observability/opentelemetry), +[OpenInference](/configure-plugins/observability/openinference), and +[Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif). ### Can I Use NeMo Relay Just For Observability Without Adaptive Tuning Or Middleware? @@ -374,8 +374,8 @@ path, or experiment. Build a plugin when the behavior should be reused across applications, activated through configuration, validated before startup, or reported through structured activation diagnostics. -Refer to [Define a Plugin](/build-plugins/basic-guide) and -[Register Plugin Behavior](/build-plugins/register-behavior). +Refer to [Language Binding Plugins](/build-plugins/language-binding/about) and +[Register Plugin Behavior](/build-plugins/language-binding/register-behavior). ### How Is Adaptive Tuning Enabled? @@ -384,10 +384,10 @@ telemetry and in-memory state so the runtime can observe representative workflows before changing behavior. Enable active behavior one area at a time, such as adaptive hints, tool parallelism, or cache-governor behavior. -Refer to [Adaptive](/adaptive-plugin/about), -[Adaptive Configuration](/adaptive-plugin/configuration), -[Adaptive Cache Governor (ACG)](/adaptive-plugin/acg), and -[Adaptive Hints](/adaptive-plugin/adaptive-hints). +Refer to [Adaptive](/configure-plugins/adaptive/about), +[Adaptive Configuration](/configure-plugins/adaptive/configuration), +[Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), and +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints). ## Framework Integration And APIs diff --git a/docs/resources/troubleshooting/index.mdx b/docs/resources/troubleshooting/index.mdx index 1a4de0f9e..d274ad92d 100644 --- a/docs/resources/troubleshooting/index.mdx +++ b/docs/resources/troubleshooting/index.mdx @@ -1,6 +1,6 @@ --- title: "Troubleshooting Guide" -description: "" +description: "Troubleshoot NeMo Relay builds, bindings, instrumentation, and runtime behavior." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -108,7 +108,7 @@ Use [Middleware](/about-nemo-relay/concepts/middleware) to confirm when an execu Confirm that the subscriber is registered before the runtime emits the events you expect. For scope-local subscribers, confirm that the active scope is the owning scope or a descendant scope. -Use [Subscribers](/about-nemo-relay/concepts/subscribers), [Events](/about-nemo-relay/concepts/events), and [Observability](/observability-plugin/about) to verify lifecycle timing and event names. +Use [Subscribers](/about-nemo-relay/concepts/subscribers), [Events](/about-nemo-relay/concepts/events), and [Observability](/configure-plugins/observability/about) to verify lifecycle timing and event names. ## Events Are Missing Expected Fields @@ -123,8 +123,8 @@ exporter subscribed after the relevant events were emitted, or the export filter does not match the active `root_uuid`. Mixed trajectories usually mean multiple agents share a root scope or the export did not filter by root scope. -Use [Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif) -and [Observability](/observability-plugin/about) to confirm exporter +Use [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) +and [Observability](/configure-plugins/observability/about) to confirm exporter setup, event collection timing, and root-scope filtering. ## LLM Stream Output Is Missing The Final Chunk @@ -143,7 +143,7 @@ Use [Non-Serializable Data](/integrate-into-frameworks/non-serializable-data), [ If a plugin fails before runtime execution, validate configuration separately from behavior registration. Check required fields, value types, defaults, and whether the plugin is reading configuration from the expected source. -Use [Validate Plugin Configuration](/build-plugins/validate-configuration), [Design Plugin Configuration](/build-plugins/advanced-configuration), and [Register Plugin Behavior](/build-plugins/register-behavior) to isolate configuration problems from runtime behavior problems. +Use [Validate Plugin Configuration](/build-plugins/language-binding/validate-configuration), [Design Plugin Configuration](/build-plugins/language-binding/advanced-configuration), and [Register Plugin Behavior](/build-plugins/language-binding/register-behavior) to isolate configuration problems from runtime behavior problems. ## Adaptive Tuning Does Not Change Behavior @@ -152,9 +152,9 @@ the runtime path actually reaches that component. If behavior does not change, check whether the configured policy is disabled, scoped too narrowly, or not connected to the call path under test. -Use [Adaptive Configuration](/adaptive-plugin/configuration), -[Adaptive Cache Governor (ACG)](/adaptive-plugin/acg), and -[Adaptive Hints](/adaptive-plugin/adaptive-hints) to verify component +Use [Adaptive Configuration](/configure-plugins/adaptive/configuration), +[Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), and +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) to verify component names and configuration scope. ## Framework Integration Behaves Differently From Core APIs diff --git a/docs/resources/troubleshooting/trace-incident-runbook.mdx b/docs/resources/troubleshooting/trace-incident-runbook.mdx index 4369c735c..dd571d4e6 100644 --- a/docs/resources/troubleshooting/trace-incident-runbook.mdx +++ b/docs/resources/troubleshooting/trace-incident-runbook.mdx @@ -1,6 +1,6 @@ --- title: "Trace Incident Runbook" -description: "" +description: "Diagnose missing, partial, duplicate, or mis-scoped NeMo Relay traces." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -122,8 +122,8 @@ already been emitted. Verify these conditions: - Shutdown, teardown, or request completion calls flush owned exporters before the process exits or the container stops. -Use [Observability](/observability-plugin/about), -[Observability Configuration](/observability-plugin/configuration), and +Use [Observability](/configure-plugins/observability/about), +[Observability Configuration](/configure-plugins/observability/configuration), and [Subscribers](/about-nemo-relay/concepts/subscribers) to verify the registration lifecycle. @@ -159,10 +159,10 @@ For OpenTelemetry or OpenInference export, confirm these settings: - The application flushes and shuts down the subscriber during graceful termination. -Refer to [Agent Trajectory Observability Format (ATOF)](/observability-plugin/atof), -[Agent Trajectory Interchange Format (ATIF)](/observability-plugin/atif), -[OpenTelemetry](/observability-plugin/opentelemetry), and -[OpenInference](/observability-plugin/openinference). +Refer to [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof), +[Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif), +[OpenTelemetry](/configure-plugins/observability/opentelemetry), and +[OpenInference](/configure-plugins/observability/openinference). ## Check For Duplicate Event Sources diff --git a/docs/supported-integrations/deepagents.mdx b/docs/supported-integrations/deepagents.mdx index 24bf37db9..39235c802 100644 --- a/docs/supported-integrations/deepagents.mdx +++ b/docs/supported-integrations/deepagents.mdx @@ -1,7 +1,7 @@ --- title: "NeMo Relay Deep Agents Integration" sidebar-title: "Deep Agents Integration Guide" -description: "" +description: "Add NeMo Relay observability to Deep Agents applications." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -91,7 +91,7 @@ when you need to capture Deep Agents skill or subagent marks. ## Verify the Integration -The integration is wired correctly when: +The integration works correctly when: - The Deep Agents run completes and prints a final response. - The `deepagents-request` scope contains the top-level agent execution. @@ -115,5 +115,5 @@ It captures: Remote graphs or processes still need NeMo Relay instrumentation in that graph or process to capture their internal model and tool calls. -Refer to [Observability](/observability-plugin/about) +Refer to [Observability](/configure-plugins/observability/about) for details on exporting NeMo Relay observability data to third-party systems. diff --git a/docs/supported-integrations/langchain.mdx b/docs/supported-integrations/langchain.mdx index 76c70b065..206ec3ae1 100644 --- a/docs/supported-integrations/langchain.mdx +++ b/docs/supported-integrations/langchain.mdx @@ -1,7 +1,7 @@ --- title: "NeMo Relay LangChain Integration" sidebar-title: "LangChain Integration Guide" -description: "" +description: "Add NeMo Relay observability to LangChain applications." position: 3 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -90,7 +90,7 @@ print(f"Final response: {final_message.content}") ## Verify the Integration -The integration is wired correctly when: +The integration works correctly when: - The agent run completes and prints a final response. - The `langchain-request` scope contains the managed model call. @@ -98,4 +98,4 @@ The integration is wired correctly when: ## Observability -Refer to [Observability](/observability-plugin/about) for details on exporting NeMo Relay observability data to third-party systems. +Refer to [Observability](/configure-plugins/observability/about) for details on exporting NeMo Relay observability data to third-party systems. diff --git a/docs/supported-integrations/langgraph.mdx b/docs/supported-integrations/langgraph.mdx index f617cc233..07eaf7a14 100644 --- a/docs/supported-integrations/langgraph.mdx +++ b/docs/supported-integrations/langgraph.mdx @@ -1,7 +1,7 @@ --- title: "NeMo Relay LangGraph Integration" sidebar-title: "LangGraph Integration Guide" -description: "" +description: "Add NeMo Relay observability to LangGraph applications." position: 4 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -103,7 +103,7 @@ pip install "nemo-relay[langgraph,langchain-nvidia]" ## Verify the Integration -The integration is wired correctly when: +The integration works correctly when: - `graph.invoke(...)` returns the incremented state from the example. - The `langgraph-request` scope contains the LangGraph run. @@ -111,4 +111,4 @@ The integration is wired correctly when: ## Observability -Refer to [Observability](/observability-plugin/about) for details on exporting NeMo Relay observability data to third-party systems. +Refer to [Observability](/configure-plugins/observability/about) for details on exporting NeMo Relay observability data to third-party systems. diff --git a/docs/supported-integrations/openclaw-plugin.mdx b/docs/supported-integrations/openclaw-plugin.mdx index 84304da69..ac758a65f 100644 --- a/docs/supported-integrations/openclaw-plugin.mdx +++ b/docs/supported-integrations/openclaw-plugin.mdx @@ -1,6 +1,6 @@ --- title: "OpenClaw Plugin Guide" -description: "" +description: "Install and configure the OpenClaw plugin for NeMo Relay observability and policy hooks." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -201,8 +201,8 @@ plugin configuration, so they use `snake_case` regardless of language. Missing observability sections are disabled. Plugin-host validation or initialization errors degrade the NeMo Relay runtime as a whole, and the status method reports configured output health from the generic observability -component. See -[Observability Configuration](/observability-plugin/configuration) +component. Refer to +[Observability Configuration](/configure-plugins/observability/configuration) for the complete `observability` component schema and exporter-specific fields. ## Verify the Integration @@ -325,6 +325,6 @@ match the same assistant turn. Current OpenClaw public hooks are separate event streams, so some LLM timing attribution is best-effort. If a matching request hook is missing, the plugin -may replay an LLM output with a placeholder request after the configured grace +can replay an LLM output with a placeholder request after the configured grace window. If timing is ambiguous, the plugin emits diagnostic marks instead of unsafe latency. diff --git a/fern/docs.yml b/fern/docs.yml index 90cf8db1f..51a92e481 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -20,6 +20,36 @@ announcement: redirects: - source: /nemo/relay/about-nemo-relay/release-notes/related-topics destination: /nemo/relay/about-nemo-relay/release-notes +- source: /nemo/relay/build-plugins/plugin-configuration-files + destination: /nemo/relay/configure-plugins/plugin-configuration-files +- source: /nemo/relay/observability-plugin/about + destination: /nemo/relay/configure-plugins/observability/about +- source: /nemo/relay/observability-plugin/configuration + destination: /nemo/relay/configure-plugins/observability/configuration +- source: /nemo/relay/observability-plugin/atif + destination: /nemo/relay/configure-plugins/observability/atif +- source: /nemo/relay/observability-plugin/atof + destination: /nemo/relay/configure-plugins/observability/atof +- source: /nemo/relay/observability-plugin/openinference + destination: /nemo/relay/configure-plugins/observability/openinference +- source: /nemo/relay/observability-plugin/opentelemetry + destination: /nemo/relay/configure-plugins/observability/opentelemetry +- source: /nemo/relay/adaptive-plugin/about + destination: /nemo/relay/configure-plugins/adaptive/about +- source: /nemo/relay/adaptive-plugin/configuration + destination: /nemo/relay/configure-plugins/adaptive/configuration +- source: /nemo/relay/adaptive-plugin/acg + destination: /nemo/relay/configure-plugins/adaptive/acg +- source: /nemo/relay/adaptive-plugin/adaptive-hints + destination: /nemo/relay/configure-plugins/adaptive/adaptive-hints +- source: /nemo/relay/nemo-guardrails-plugin/about + destination: /nemo/relay/configure-plugins/nemo-guardrails/about +- source: /nemo/relay/nemo-guardrails-plugin/configuration + destination: /nemo/relay/configure-plugins/nemo-guardrails/configuration +- source: /nemo/relay/pii-redaction-plugin/about + destination: /nemo/relay/configure-plugins/pii-redaction/about +- source: /nemo/relay/pii-redaction-plugin/configuration + destination: /nemo/relay/configure-plugins/pii-redaction/configuration experimental: mdx-components: From 1b610f297c0cc9293fa2ab4bb6f0764bae042cb0 Mon Sep 17 00:00:00 2001 From: Will Killian <2007799+willkill07@users.noreply.github.com> Date: Tue, 7 Jul 2026 22:45:23 -0400 Subject: [PATCH 2/4] docs: update 0.5 release notes (#362) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #### Overview Update the documentation-site release notes for NVIDIA NeMo Relay 0.5 and add a maintainer skill that gathers release evidence from Git history. - [x] I confirm this contribution is my own work, or I have the right to submit it under the project license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Replace the stale 0.4 release-note content with the 0.5 compatibility migrations, dynamic plugin capabilities, CLI diagnostics, and runtime and observability updates. - Preserve the complete fixed-item history recorded in the 0.3 and 0.4 release-note pages under release-labelled sections. - Add `draft-release-notes`, including a read-only Git evidence collector that compares release-note trees and groups commits for maintainer review. #### Where should the reviewer start? Start with `docs/about-nemo-relay/release-notes/known-issues.mdx` for the complete prior-release fix history, then review `.agents/skills/draft-release-notes/SKILL.md` and its evidence collector. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none ## Summary by CodeRabbit * **Documentation** * Updated NeMo Relay release notes from **0.4** to **0.5**, including highlights, release scope/capabilities, and compatibility/migration changes. * Refreshed **known issues** and “Fixed in” notes for NeMo Relay **0.5**, plus reorganized prior-release sections. * **New Features** * Added a guided **Draft Release Notes** workflow to standardize evidence-based authoring, claim validation, and targeted MDX updates. * Introduced an evidence-collection helper that summarizes Git/doc changes between release references for use during drafting and review. Authors: - Will Killian (https://github.com/willkill07) Approvers: - Bryan Bednarski (https://github.com/bbednarski9) - https://github.com/lvojtku URL: https://github.com/NVIDIA/NeMo-Relay/pull/362 --- .agents/skills/draft-release-notes/SKILL.md | 65 ++++++ .../scripts/collect_release_evidence.py | 187 ++++++++++++++++++ .../release-notes/highlights.mdx | 128 ++++++------ docs/about-nemo-relay/release-notes/index.mdx | 73 ++++--- .../release-notes/known-issues.mdx | 84 +++++--- 5 files changed, 397 insertions(+), 140 deletions(-) create mode 100644 .agents/skills/draft-release-notes/SKILL.md create mode 100755 .agents/skills/draft-release-notes/scripts/collect_release_evidence.py diff --git a/.agents/skills/draft-release-notes/SKILL.md b/.agents/skills/draft-release-notes/SKILL.md new file mode 100644 index 000000000..7e9e37c74 --- /dev/null +++ b/.agents/skills/draft-release-notes/SKILL.md @@ -0,0 +1,65 @@ +--- +name: draft-release-notes +description: Compare NeMo Relay release branches and draft or update the documentation-site release notes. Use when preparing a release-notes update under docs/about-nemo-relay/release-notes, reviewing release-to-release changes, or gathering evidence for a current release summary. +--- + +# Draft Release Notes + +Draft the three documentation-site release-notes pages from verified repository +evidence. Keep complete, PR-by-PR release history in GitHub Releases. + +## Gather Evidence + +Run the read-only helper with explicit release refs and the target minor version: + +```bash +python3 .agents/skills/draft-release-notes/scripts/collect_release_evidence.py \ + --previous release/. \ + --current HEAD \ + --version . +``` + +The report verifies both refs, compares their release-notes trees, identifies +version text currently present in those pages, and groups commits into review +candidates. Treat the groups as an evidence index, not publication-ready copy. + +## Workflow + +1. Confirm the target release version from the release branch and package + metadata. Preserve unrelated working-tree changes. +2. Run the helper. It reports an absent prior release-notes directory without + failing, which is expected for early release branches. +3. Verify each candidate claim in the changed public docs, API types, command + help, or source before including it. Prioritize breaking changes, migrations, + user-visible features, and ongoing support limitations. +4. Update only these pages unless the release changes their route or entry + points: + - `docs/about-nemo-relay/release-notes/index.mdx` + - `docs/about-nemo-relay/release-notes/highlights.mdx` + - `docs/about-nemo-relay/release-notes/known-issues.mdx` +5. Keep the existing page roles: + - `index.mdx` gives the current-release summary, compatibility notes, scope, + and curated feature links. + - `highlights.mdx` groups notable changes by user-facing theme. + - `known-issues.mdx` records current limitations, compatibility migrations, + current-release fixes, and the complete fixed-item history recorded in + earlier release-note pages. Preserve every prior fixed-item bullet under + release-labelled subsections; do not summarize, deduplicate, or omit it. + GitHub Releases remains the complete PR-by-PR history. +6. Preserve MDX front matter and the JSX SPDX comment. State the full history + is available in GitHub Releases; do not create a changelog or GitHub Release + body from this skill. + +## Validate + +Run the helper for the target release and for an early branch without release +notes when available. Review public claims, then run: + +```bash +git diff --check +just docs +just docs-linkcheck +``` + +Check product names, commands, package names, support claims, and links against +the current repository before handing off the draft. diff --git a/.agents/skills/draft-release-notes/scripts/collect_release_evidence.py b/.agents/skills/draft-release-notes/scripts/collect_release_evidence.py new file mode 100755 index 000000000..f385fdf1a --- /dev/null +++ b/.agents/skills/draft-release-notes/scripts/collect_release_evidence.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Collect read-only Git evidence for NeMo Relay documentation release notes.""" + +from __future__ import annotations + +import argparse +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path + +RELEASE_NOTES_DIR = "docs/about-nemo-relay/release-notes" +VERSION_PATTERN = re.compile(r"(?:NVIDIA )?NeMo Relay (\d+\.\d+(?:\.\d+)?)") + + +@dataclass(frozen=True) +class Commit: + """One non-merge commit and its changed paths.""" + + sha: str + subject: str + paths: tuple[str, ...] + + +def run_git(repo: Path, *args: str) -> str: + """Run Git without a shell and return stdout or raise a readable error.""" + result = subprocess.run( + ["git", *args], + cwd=repo, + check=False, + capture_output=True, + encoding="utf-8", + ) + if result.returncode: + detail = result.stderr.strip() or result.stdout.strip() + raise RuntimeError(f"git {' '.join(args)} failed: {detail}") + return result.stdout + + +def resolve_ref(repo: Path, ref: str) -> str: + """Return the commit ID for a user-supplied ref.""" + return run_git(repo, "rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}").strip() + + +def note_files(repo: Path, ref: str) -> tuple[str, ...]: + """List release-note files that exist at a ref.""" + output = run_git(repo, "ls-tree", "-r", "--name-only", ref, "--", RELEASE_NOTES_DIR) + return tuple(path for path in output.splitlines() if path) + + +def note_versions(repo: Path, ref: str, paths: tuple[str, ...]) -> tuple[str, ...]: + """Return the NeMo Relay versions mentioned in release-note files.""" + versions: set[str] = set() + for path in paths: + content = run_git(repo, "show", f"{ref}:{path}") + versions.update(VERSION_PATTERN.findall(content)) + return tuple(sorted(versions)) + + +def commits(repo: Path, previous: str, current: str) -> tuple[Commit, ...]: + """Read non-merge commits and their changed paths, newest first.""" + output = run_git(repo, "log", "--no-merges", "--format=%H%x09%s", f"{previous}..{current}") + collected: list[Commit] = [] + for line in output.splitlines(): + sha, subject = line.split("\t", maxsplit=1) + names = run_git(repo, "diff-tree", "--no-commit-id", "--name-status", "-r", sha) + paths = tuple(entry.rsplit("\t", maxsplit=1)[-1] for entry in names.splitlines() if entry) + collected.append(Commit(sha=sha, subject=subject, paths=paths)) + return tuple(collected) + + +def category(commit: Commit) -> str: + """Classify a commit for review without treating the result as a release claim.""" + subject = commit.subject.lower() + if "!" in commit.subject.split(":", maxsplit=1)[0] or re.search( + r"\b(remove|drop|breaking|deprecat|migration)\b", subject + ): + return "Breaking and migration candidates" + if subject.startswith("feat"): + return "Feature candidates" + if subject.startswith(("fix", "perf")): + return "Fix candidates" + return "Documentation and tooling candidates" + + +def public_paths(commits_to_report: tuple[Commit, ...]) -> tuple[str, ...]: + """Return changed paths likely to support public release-note claims.""" + package_paths = { + "Cargo.toml", + "crates/node/package.json", + "integrations/openclaw/package.json", + "python/plugin/pyproject.toml", + } + return tuple( + sorted( + { + path + for commit in commits_to_report + for path in commit.paths + if path == "README.md" + or path.startswith("docs/") + or path.endswith("/README.md") + or path in package_paths + } + ) + ) + + +def print_note_tree(repo: Path, label: str, ref: str) -> None: + """Print structure and version mentions for one release ref.""" + paths = note_files(repo, ref) + print(f"## {label} Release-Notes Tree") + if not paths: + print(f"No files exist under `{RELEASE_NOTES_DIR}` at `{ref}`.\n") + return + print("Files:") + for path in paths: + print(f"- `{path}`") + versions = note_versions(repo, ref, paths) + description = ", ".join(versions) if versions else "no NeMo Relay version text detected" + print(f"Version text: {description}\n") + + +def main() -> int: + """Parse arguments and print the evidence report.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--previous", required=True, help="Previous release branch or commit") + parser.add_argument("--current", required=True, help="Current release branch or commit") + parser.add_argument("--version", required=True, help="Target major.minor release version") + parser.add_argument("--repo", default=".", help="Repository root (default: current directory)") + args = parser.parse_args() + + repo = Path(args.repo).resolve() + try: + previous_sha = resolve_ref(repo, args.previous) + current_sha = resolve_ref(repo, args.current) + run_git(repo, "merge-base", "--is-ancestor", previous_sha, current_sha) + except RuntimeError as error: + parser.error(str(error)) + + print(f"# Release-Note Evidence: {args.version}") + print() + print(f"- Previous: `{args.previous}` ({previous_sha[:12]})") + print(f"- Current: `{args.current}` ({current_sha[:12]})") + print(f"- Merge base: `{run_git(repo, 'merge-base', previous_sha, current_sha).strip()[:12]}`") + print() + print_note_tree(repo, "Previous", args.previous) + print_note_tree(repo, "Current", args.current) + + changes = commits(repo, previous_sha, current_sha) + grouped: dict[str, list[Commit]] = { + "Breaking and migration candidates": [], + "Feature candidates": [], + "Fix candidates": [], + "Documentation and tooling candidates": [], + } + for commit in changes: + grouped[category(commit)].append(commit) + + print("## Commit Candidates") + for heading, entries in grouped.items(): + print(f"### {heading}") + if not entries: + print("- None") + for commit in entries: + paths = ", ".join(f"`{path}`" for path in commit.paths[:4]) + suffix = f" — {paths}" if paths else "" + print(f"- `{commit.sha[:12]}` {commit.subject}{suffix}") + print() + + print("## Changed Public Documentation and Package Surfaces") + paths = public_paths(changes) + if paths: + for path in paths: + print(f"- `{path}`") + else: + print("- None") + print() + print("Review every candidate against current public docs and APIs before drafting prose.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/about-nemo-relay/release-notes/highlights.mdx b/docs/about-nemo-relay/release-notes/highlights.mdx index c32b5ad24..2da2b67f7 100644 --- a/docs/about-nemo-relay/release-notes/highlights.mdx +++ b/docs/about-nemo-relay/release-notes/highlights.mdx @@ -9,75 +9,65 @@ SPDX-License-Identifier: Apache-2.0 */} This page summarizes the notable capabilities in the current release documentation set. -## NVIDIA NeMo Relay 0.4 - -NVIDIA NeMo Relay 0.4 adds host plugin installation, first-party PII redaction, local -Guardrails execution, pricing-aware observability, streaming raw-event export, -and stronger coding-agent trace fidelity. - -### Compatibility Notes - -- The Python-only NeMo Guardrails example plugin was removed. Applications - should use the built-in `nemo_guardrails` component instead. -- Pricing estimates require an inline, file-backed, or embedding-provided - pricing source. NeMo Relay does not ship provider price data by default. -- Local NeMo Guardrails mode requires Python 3.11 or newer and - `nemoguardrails==0.22.0` in the runtime environment. - -### CLI And Plugin Configuration - -- Added `nemo-relay install`, `uninstall`, and `doctor --plugin` flows for - Claude Code and Codex host plugins. -- Added generated local marketplace support so Claude Code and Codex can load - NeMo Relay through native host plugin mechanisms. -- Layered code-driven plugin configuration over materialized global, project, - and user plugin files while preserving documented file precedence. -- Added pricing source management commands for validating catalogs, adding - file-backed sources, and resolving model pricing. - -### Guardrails And Redaction - -- Added a local Python-backed backend for the built-in `nemo_guardrails` - component. -- Expanded Guardrails configuration docs for local and remote backend behavior, - supported codecs, request defaults, tool boundaries, and streaming behavior. -- Added the first-party PII redaction plugin crate with deterministic local - backend support and Python and Node.js helper surfaces. - -### Observability - -- Added ATOF streaming endpoints for HTTP POST, WebSocket, and long-lived - NDJSON uploads. -- Added ATIF HTTP storage export alongside S3-compatible storage destinations. -- Added model-pricing lookup and cost layering for managed LLM responses. -- Propagated normalized LLM cost into ATIF step metrics, OpenInference - attributes, and OpenTelemetry attributes. -- Set OpenTelemetry span status from NeMo Relay execution results. -- Preserved sanitized LLM request payloads in annotations and start-event - output. - -### Integrations - -- Improved Hermes hook injection, routed-provider observability, wrapped ATIF - fidelity, subagent lineage, and error-path export consistency. -- Added Codex observability contract coverage and suppressed noisy Claude Code - lifecycle events. -- Added nested subagent session lineage and more consistent cost, provenance, - replay, placeholder, and hook-only fallback exports for OpenClaw. -- Improved LangChain input serialization and fixed plugin context-manager - deadlock behavior. -- Added flattened OpenInference LLM attributes for annotations and replay. -- Annotated Deep Agents model responses and refreshed LangGraph callback - coverage. - -### Documentation And Tooling - -- Updated installation documentation for 0.4 packages and host plugin setup. -- Clarified Hermes integration paths and coding-agent plugin source manifests. -- Added NVSkills CI request workflow and regenerated NeMo Relay skill eval - datasets. -- Updated dependency attribution files and isolated Rust test behavior for CI - and SonarQube cleanup. +## NVIDIA NeMo Relay 0.5 + +NVIDIA NeMo Relay 0.5 adds dynamic plugin SDKs and controls, verified CLI +installation and diagnostics, clearer runtime contracts, and improved lineage +for managed LLM requests. + +### Compatibility and Migration + +- Move plugin configuration from `config.toml` into `plugins.toml`. The + `--plugin-config` CLI option and hook-forwarding input have been removed. +- Update every registered LLM request and tool execution intercept to return + its canonical outcome. The outcomes can include ordered pending marks while + Relay retains lifecycle ownership. +- Rebuild development native plugins and `grpc-v1` workers against 0.5 because + the native ABI and worker outcome contracts are now finalized. +- Migrate away from the removed experimental `nemo-relay-wasm` package, Cursor + CLI support, and patch-based integrations. + +### Dynamic Plugins and Configuration + +- Added dynamic plugin discovery from `plugins.toml`, host policy and + attestation gates, and lifecycle commands for inspecting, validating, + enabling, and disabling plugins. +- Added the Rust SDK for trusted in-process `rust_dynamic` plugins and the + `grpc-v1` worker protocol with Rust and Python worker SDKs. +- Added schema-aware dynamic-plugin configuration, validation, secret-aware + editing, and merged TOML preview in the CLI. +- Added Python worker-environment management and worker invocation + cancellation. + +### CLI Installation and Diagnostics + +- Added verified Unix and Windows installers that select the release asset, + verify its SHA-256 checksum, and support version pinning. +- Expanded `nemo-relay doctor` with agent-host readiness checks, persistent + host-plugin diagnostics, and discovered `plugins.toml` source and validation + reporting. +- Added gateway status reporting for bind, exporter, and remote ATIF storage + destinations. + +### Runtime and Observability + +- Added canonical LLM request and tool execution outcomes across Rust, Python, + Node.js, Go, C FFI, native plugins, and worker plugins. +- Added ordered pending marks to managed LLM and tool lifecycles without adding + control data to application-visible requests or tool results. +- Added automatic Dynamo session and parent-session headers to managed LLM + requests, plus bounded CLI turn scopes for trace correlation. +- Documented stable normalized token and cost semantics, including provider + mapping and exporter projection behavior. +- Hardened ATOF HTTP output and ensured gateway shutdown flushes queued events. + +### Documentation and Tooling + +- Added public contract references for LLM request and tool execution intercept + outcomes. +- Expanded dynamic plugin, configuration, runtime-contract, installation, and + integration examples. +- Updated package, build, test, and release guidance for the primary Rust, Python, and Node.js surfaces, plus experimental Go and raw C FFI notes. The complete changelog and release artifacts can be viewed on [GitHub Releases](https://github.com/NVIDIA/NeMo-Relay/releases). diff --git a/docs/about-nemo-relay/release-notes/index.mdx b/docs/about-nemo-relay/release-notes/index.mdx index 9ae766dbf..c275fa7c1 100644 --- a/docs/about-nemo-relay/release-notes/index.mdx +++ b/docs/about-nemo-relay/release-notes/index.mdx @@ -12,23 +12,21 @@ tag-specific notes. ## Current Release -NeMo Relay 0.4 focuses on host plugin installation, first-party PII redaction, -local NeMo Guardrails execution, pricing-aware LLM response annotations, -streaming raw ATOF export, remote ATIF storage, and stronger coding-agent trace -fidelity. +NVIDIA NeMo Relay 0.5 adds dynamic plugin SDKs and lifecycle controls, verified +CLI installers and readiness diagnostics, stable token and cost field semantics, +and stronger request and trace lineage behavior. The most important compatibility notes are: -- The Python-only NeMo Guardrails example plugin has been removed. Use the - built-in `nemo_guardrails` component for supported Guardrails integration. -- Pricing estimates require configured pricing sources. NeMo Relay does not - ship provider price data by default. -- Local NeMo Guardrails mode depends on Python 3.11 or newer and an available - `nemoguardrails` runtime. -- Node.js 24 or newer remains the minimum supported Node.js version. -- Native subscriber delivery remains asynchronous. Code that depends on - subscriber callback side effects, exporter output, or deterministic test - assertions must call the subscriber flush API. +- Plugin configuration now lives in `plugins.toml`. Move plugin settings out of + `config.toml`; the `--plugin-config` CLI option has been removed. +- Registered LLM request and tool execution intercepts now return canonical + outcomes. Rebuild development native plugins and `grpc-v1` workers against + 0.5. +- The experimental `nemo-relay-wasm` package, Cursor CLI support, and + patch-based integrations have been removed. +- Go and the raw C FFI remain experimental and source-first. Node.js 24 or + newer remains the minimum supported Node.js version. Use the child pages for release highlights and support notes. For the complete PR-by-PR changelog, release artifacts, and tag-specific history, use @@ -38,32 +36,29 @@ PR-by-PR changelog, release artifacts, and tag-specific history, use This release includes: -- Built-in NeMo Guardrails plugin support, including remote and local - Python-backed backends. -- CLI installation, uninstallation, and doctor flows for Claude Code and Codex - host plugins. -- Code-driven plugin configuration layered over materialized global, project, - and user plugin files. -- First-party PII redaction plugin helpers across Rust, Python, and Node.js. -- Model-pricing catalogs and LLM cost annotations that propagate into ATIF, - OpenInference, and OpenTelemetry output. -- ATOF streaming endpoints for HTTP POST, WebSocket, and NDJSON collectors. -- ATIF remote storage support for HTTP endpoints in addition to S3-compatible - storage. -- Trace fidelity fixes for Hermes, OpenClaw, LangChain, LangGraph, and Deep - Agents integrations. +- Dynamic plugin discovery, trust policy and attestation gates, and lifecycle + controls through the `nemo-relay` CLI. +- A Rust native plugin SDK for trusted in-process `rust_dynamic` plugins, plus + `grpc-v1` worker support through Rust and Python SDKs. +- Schema-aware dynamic-plugin configuration and editing through `plugins.toml`. +- Verified Unix and Windows CLI installers, and expanded `nemo-relay doctor` + readiness diagnostics for agent hosts and plugin configuration. +- Canonical LLM request and tool execution intercept outcomes with ordered + pending marks across the runtime and supported bindings. +- Automatic Dynamo session-lineage headers on managed LLM requests and + turn-scoped CLI trace correlation. +- Stable normalized token and cost semantics for provider response codecs and + ATOF, ATIF, OpenInference, and OpenTelemetry projections. ## Feature Documentation -For the major 0.4 additions, start with: +For the major 0.5 additions, start with: -- [Plugin Installation](/nemo-relay-cli/plugin-installation) -- [Observability Plugin](/configure-plugins/observability/about) -- [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) -- [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) -- [Provider Response Codecs And Pricing](/integrate-into-frameworks/provider-response-codecs) -- [NeMo Guardrails Plugin](/configure-plugins/nemo-guardrails/about) -- [PII Redaction API Reference](/reference/api/python-library-reference/pii-redaction) -- [OpenClaw Plugin Guide](/supported-integrations/openclaw-plugin) -- [Hermes CLI Guide](/nemo-relay-cli/hermes) -- [LangChain Integration Guide](/supported-integrations/langchain) +- [Build Plugins](/build-plugins/about) +- [Plugin Configuration Files](/build-plugins/plugin-configuration-files) +- [Installation](/getting-started/installation) +- [Plugin Installation and Diagnostics](/nemo-relay-cli/plugin-installation) +- [LLM Request Intercept Outcomes](/reference/llm-request-intercept-outcomes) +- [Tool Execution Intercept Outcomes](/reference/tool-execution-intercept-outcomes) +- [Instrument an LLM Call](/instrument-applications/instrument-llm-call) +- [Provider Response Codecs and Model Pricing](/integrate-into-frameworks/provider-response-codecs) diff --git a/docs/about-nemo-relay/release-notes/known-issues.mdx b/docs/about-nemo-relay/release-notes/known-issues.mdx index 556ea425b..7d1b64a4d 100644 --- a/docs/about-nemo-relay/release-notes/known-issues.mdx +++ b/docs/about-nemo-relay/release-notes/known-issues.mdx @@ -9,20 +9,16 @@ SPDX-License-Identifier: Apache-2.0 */} This page lists current limitations and support notes for the release documentation set. -## NeMo Relay 0.4 +## NVIDIA NeMo Relay 0.5 -These notes apply to the NeMo Relay 0.4 release. The following known issues -and limitations apply to NeMo Relay 0.4: +These notes apply to the NVIDIA NeMo Relay 0.5 release. -- Go and the raw C FFI surface are experimental and source-first. -- Generated API pages cover Rust, Python, and Node.js. Experimental bindings do - not yet have the same generated documentation depth. -- The NeMo Relay CLI is experimental. Coding agent observability support varies - due to capabilities of hooks. Any encountered problems should be filed as - bugs. -- Host plugin mode depends on each coding-agent host's plugin, hook, and - provider-routing behavior. Hooks alone cannot produce complete LLM request - and response spans. +- Go and the raw C FFI surface are experimental and source-first. Generated + API pages cover Rust, Python, and Node.js; experimental bindings do not have + the same generated documentation depth. +- The NeMo Relay CLI is experimental. Coding-agent observability support varies + with host plugin, hook, and provider-routing capabilities. Hooks alone cannot + produce complete LLM request and response spans. - Complete first-request capture in Codex plugin mode depends on Codex firing an installed hook before the first provider request. - Node.js 24 or newer is required for Node.js binding and package workflows. @@ -30,33 +26,53 @@ and limitations apply to NeMo Relay 0.4: optimization support. Security is limited to pre-tool conditional guardrails, and optimization is limited to adaptive telemetry unless the integration owns a managed execution path. -- The NeMo Guardrails plugin remote backend depends on the availability, - latency, and policy behavior of the configured remote service. -- The NeMo Guardrails plugin local backend starts a Python worker subprocess. - That worker environment must provide Python 3.11 or newer and - `nemoguardrails==0.22.0`; `SUPPORTED_NEMOGUARDRAILS_VERSION` in - `crates/core/src/plugins/nemo_guardrails/local_worker.py` is the - authoritative pin. +- The NeMo Guardrails remote backend depends on the availability, latency, and + policy behavior of the configured remote service. Its local backend requires + Python 3.11 or newer and `nemoguardrails==0.22.0`. - The PII redaction plugin currently provides deterministic local backend support. Local-model backend configuration is reserved for future expansion. -- Pricing estimates depend on configured pricing sources and the freshness of - the source catalog. Unknown model pricing and missing token data leave cost - absent instead of defaulting to zero. -- ATOF streaming endpoints depend on collector availability. Failed endpoints - are skipped or retried without blocking file output or other configured - endpoints. -- S3-compatible ATIF export requires valid storage credentials and endpoint - configuration in the runtime environment. -- Remote ATIF storage requires valid destination credentials and endpoint - configuration in the runtime environment. +- Pricing estimates depend on configured model-pricing sources and their + freshness. Unknown model pricing and missing token data leave cost absent + instead of defaulting to zero. +- ATOF streaming endpoints and remote ATIF storage depend on reachable, + correctly configured destinations. Endpoint failures do not block local file + output or other configured endpoints. +- Native dynamic plugins run in-process and are not sandboxed. `grpc-v1` worker + plugins isolate a process but do not provide a security sandbox. - `LLMRequest` objects in the Python binding should be treated as immutable. Request middleware that changes content should return a new request object. - Native subscriber callbacks are delivered asynchronously. Flush subscribers before relying on callback side effects, captured event lists, files, or exporter output. Deregistering a subscriber affects future emissions, but - callbacks from already-queued event snapshots may still run. + callbacks from already-queued event snapshots can still run. -### Fixed in NeMo Relay 0.4 +### Compatibility and Migration Notes + +- Move plugin configuration from `config.toml` to `plugins.toml`. The + `--plugin-config` option is unavailable in 0.5. +- Registered LLM request and tool execution intercepts must return their + canonical outcome types. Rebuild development native plugins and `grpc-v1` + workers against 0.5. +- The experimental `nemo-relay-wasm` package is no longer built, published, or + supported by this repository. +- The `nemo-relay cursor` and `hook-forward cursor` entry points are no longer + available. Patch-based integrations are also no longer maintained. + +### Fixed in NVIDIA NeMo Relay 0.5 + +- Gateway shutdown flushes queued events before it exits. +- Dynamic worker invocations support cancellation, and the CLI manages Python + dynamic-plugin environments. +- ATOF file export creates a missing output directory before opening JSONL + output. +- CLI status output reports daemon binding, exporter status, and remote ATIF + storage destinations more reliably. +- Codex host-plugin discovery uses the supported text output from the Codex + plugin command. + +### Fixed in Prior Releases + +#### NVIDIA NeMo Relay 0.4 - ATIF shutdown no longer deadlocks queued subscribers. - Sanitized LLM requests are resolved from annotations for observability output. @@ -72,11 +88,15 @@ and limitations apply to NeMo Relay 0.4: - Node.js `withScope` callbacks receive a real `ScopeHandle`. - Deep Agents model responses are annotated for downstream observability. -### Fixed in Earlier Releases +#### NVIDIA NeMo Relay 0.3 - Managed LLM start events are emitted before execution intercepts. - Coding-agent trace scopes are aligned with NeMo Relay agent scope semantics. - ATIF tool observations are correlated with their matching tool calls. - OpenClaw tool call replay visibility is preserved. +- LangChain serialization handles wrapped integration payloads more reliably. + +#### Earlier Releases + - Enabled TLS support for OTLP HTTP export. - Preserved Go scope stacks across OS threads. From dd55c28046f59328df48c1c2a837cbf9fa58e231 Mon Sep 17 00:00:00 2001 From: Will Killian <2007799+willkill07@users.noreply.github.com> Date: Wed, 8 Jul 2026 09:31:54 -0400 Subject: [PATCH 3/4] fix: generate references for published packages (#378) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #### Overview Generate complete API references for published NeMo Relay Rust crates and Python packages. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Include the published types, native plugin, worker protocol, and worker SDK crates in Rust reference generation. - Generate package-scoped Python references for both `nemo-relay` and `nemo-relay-plugin`; the latter renders its public worker SDK symbols. - Redirect the legacy PII Python-reference URL to its package-scoped location. #### Where should the reviewer start? Start with `scripts/docs/generate_python_library_reference.py` and `scripts/docs/generate_rust_library_reference.py`, which declare the generated package and crate surfaces. Validation: `just docs`; `uv run pre-commit run --all-files`. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none ## Summary by CodeRabbit * **New Features** * Enhanced the Python API reference generator to support multiple packages with package-aware URLs, navigation, and landing pages. * Expanded the Rust API reference generation to cover additional NeMo Relay crates. * **Documentation** * Updated documentation redirect rules, including restoration of the release-notes “related topics” redirect and added new generated API reference redirects, plus build plugin guide and CLI “about” redirects. * Updated the “Plugin Configuration Files” documentation route to the new path. * **Bug Fixes** * Fixed legacy documentation redirects (including the `pii-redaction` destination). Signed-off-by: Will Killian --- docs/about-nemo-relay/release-notes/index.mdx | 2 +- fern/docs.yml | 31 +++++ .../docs/generate_python_library_reference.py | 127 +++++++++++------- .../docs/generate_rust_library_reference.py | 22 ++- 4 files changed, 122 insertions(+), 60 deletions(-) diff --git a/docs/about-nemo-relay/release-notes/index.mdx b/docs/about-nemo-relay/release-notes/index.mdx index c275fa7c1..48e65a6af 100644 --- a/docs/about-nemo-relay/release-notes/index.mdx +++ b/docs/about-nemo-relay/release-notes/index.mdx @@ -55,7 +55,7 @@ This release includes: For the major 0.5 additions, start with: - [Build Plugins](/build-plugins/about) -- [Plugin Configuration Files](/build-plugins/plugin-configuration-files) +- [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) - [Installation](/getting-started/installation) - [Plugin Installation and Diagnostics](/nemo-relay-cli/plugin-installation) - [LLM Request Intercept Outcomes](/reference/llm-request-intercept-outcomes) diff --git a/fern/docs.yml b/fern/docs.yml index 51a92e481..4ad62a151 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -18,8 +18,25 @@ announcement: message: "🔔 NVIDIA NeMo Relay is beta software. APIs and behavior may change with each release." redirects: +# Release notes - source: /nemo/relay/about-nemo-relay/release-notes/related-topics destination: /nemo/relay/about-nemo-relay/release-notes + +# Generated API references +- source: /nemo/relay/reference/api/python-library-reference/pii-redaction + destination: /nemo/relay/reference/api/python-library-reference/nemo-relay/pii-redaction +- source: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/pricing + destination: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/model-pricing +- source: /nemo/relay/reference/api/rust-library-reference/nemo-relay/plugins/pricing + destination: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/model-pricing +- source: /nemo/relay/reference/api/python-library-reference/:slug* + destination: /nemo/relay/reference/api/python-library-reference/nemo-relay/:slug* +- source: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/pricing/:slug* + destination: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/model-pricing/:slug* +- source: /nemo/relay/reference/api/rust-library-reference/nemo-relay/plugins/pricing/:slug* + destination: /nemo/relay/reference/api/rust-library-reference/nemo-relay/codec/model-pricing/:slug* + +# Plugin configuration documentation - source: /nemo/relay/build-plugins/plugin-configuration-files destination: /nemo/relay/configure-plugins/plugin-configuration-files - source: /nemo/relay/observability-plugin/about @@ -51,6 +68,20 @@ redirects: - source: /nemo/relay/pii-redaction-plugin/configuration destination: /nemo/relay/configure-plugins/pii-redaction/configuration +# Legacy release documentation +- source: /nemo/relay/v0.3.0/build-plugins/nemoguardrails + destination: /nemo/relay/configure-plugins/nemo-guardrails/about + +# Build plugin guides +- source: /nemo/relay/build-plugins/basic-guide + destination: /nemo/relay/configure-plugins/language-binding/about +- source: /nemo/relay/build-plugins/:slug* + destination: /nemo/relay/configure-plugins/language-binding/:slug* + +# CLI guides +- source: /nemo/relay/nemo-relay-cli/cursor + destination: /nemo/relay/nemo-relay-cli/about + experimental: mdx-components: - ./components diff --git a/scripts/docs/generate_python_library_reference.py b/scripts/docs/generate_python_library_reference.py index 411010e77..e440dedc5 100644 --- a/scripts/docs/generate_python_library_reference.py +++ b/scripts/docs/generate_python_library_reference.py @@ -16,6 +16,32 @@ BASE_URL = "/reference/api/python-library-reference" +@dataclass(frozen=True, slots=True) +class Package: + distribution_name: str + module_name: str + source_root: Path + description: str + public_api_source: Path | None = None + + +PACKAGES = ( + Package( + distribution_name="nemo-relay", + module_name="nemo_relay", + source_root=Path("python/nemo_relay"), + description="Python runtime SDK for NeMo Relay.", + ), + Package( + distribution_name="nemo-relay-plugin", + module_name="nemo_relay_plugin", + source_root=Path("python/plugin/src/nemo_relay_plugin"), + description="Python SDK for NeMo Relay dynamic worker plugins.", + public_api_source=Path("_api.py"), + ), +) + + @dataclass(slots=True) class FunctionDoc: name: str @@ -34,6 +60,7 @@ class ClassDoc: @dataclass(slots=True) class ModuleDoc: + package: Package name: str title: str source: Path @@ -52,12 +79,12 @@ def _is_public(name: str) -> bool: return not name.startswith("_") -def _module_name(package_root: Path, path: Path) -> str: +def _module_name(package: Package, package_root: Path, path: Path) -> str: rel = path.relative_to(package_root) parts = list(rel.with_suffix("").parts) if parts[-1] == "__init__": parts.pop() - return ".".join([package_root.name, *parts]) + return ".".join([package.module_name, *parts]) def _is_public_module(package_root: Path, path: Path) -> bool: @@ -211,12 +238,20 @@ def _function_doc( ) -def _collect_module(package_root: Path, path: Path, api_tree: ast.Module, impl_tree: ast.Module | None) -> ModuleDoc: +def _collect_module( + package: Package, + package_root: Path, + module_path: Path, + source_path: Path, + api_tree: ast.Module, + impl_tree: ast.Module | None, +) -> ModuleDoc: impl_nodes = _impl_lookup(impl_tree) module = ModuleDoc( - name=_module_name(package_root, path), - title=_module_name(package_root, path), - source=path, + package=package, + name=_module_name(package, package_root, module_path), + title=_module_name(package, package_root, module_path), + source=source_path, summary=_summary((ast.get_docstring(impl_tree) if impl_tree else None) or ast.get_docstring(api_tree)), ) @@ -234,7 +269,7 @@ def _collect_module(package_root: Path, path: Path, api_tree: ast.Module, impl_t ) elif isinstance(node, (ast.Assign, ast.AnnAssign)): module.aliases.extend(_assignment_names(node)) - elif path.stem == "__init__" and isinstance(node, ast.ImportFrom): + elif module_path.stem == "__init__" and isinstance(node, ast.ImportFrom): module.reexports.extend(_reexport_names(node)) module.aliases = sorted(set(module.aliases)) @@ -242,7 +277,7 @@ def _collect_module(package_root: Path, path: Path, api_tree: ast.Module, impl_t return module -def _discover_modules(package_root: Path) -> list[ModuleDoc]: +def _discover_modules(package: Package, package_root: Path) -> list[ModuleDoc]: py_files = {path.with_suffix("").relative_to(package_root): path for path in package_root.rglob("*.py")} pyi_files = {path.with_suffix("").relative_to(package_root): path for path in package_root.rglob("*.pyi")} keys = sorted(set(py_files) | set(pyi_files), key=lambda key: (len(key.parts), key.parts)) @@ -255,9 +290,14 @@ def _discover_modules(package_root: Path) -> list[ModuleDoc]: continue if api_path.name == "_native.pyi": continue - api_tree = _parse(api_path) - impl_tree = _parse(impl_path) if impl_path and impl_path != api_path else None - modules.append(_collect_module(package_root, api_path, api_tree, impl_tree)) + source_path = ( + package_root / package.public_api_source + if key == Path("__init__") and package.public_api_source is not None + else api_path + ) + api_tree = _parse(source_path) + impl_tree = _parse(impl_path) if impl_path and impl_path != source_path else None + modules.append(_collect_module(package, package_root, api_path, source_path, api_tree, impl_tree)) return modules @@ -281,6 +321,10 @@ def _leaf_title(module: ModuleDoc) -> str: return _module_parts(module)[-1] +def _package_slug(package: Package) -> str: + return package.distribution_name + + def _module_has_children(module: ModuleDoc, modules: list[ModuleDoc]) -> bool: parts = _module_parts(module) return any( @@ -290,16 +334,16 @@ def _module_has_children(module: ModuleDoc, modules: list[ModuleDoc]) -> bool: def _module_url(module: ModuleDoc) -> str: - return f"{BASE_URL}/{'/'.join(_nav_parts(module))}" + return f"{BASE_URL}/{'/'.join((_package_slug(module.package), *_nav_parts(module)))}" def _module_output_path(output_dir: Path, module: ModuleDoc, modules: list[ModuleDoc]) -> Path: nav_parts = _nav_parts(module) if len(_module_parts(module)) == 1: - return output_dir / f"{nav_parts[0]}.mdx" + return output_dir / _package_slug(module.package) / "index.mdx" if _module_has_children(module, modules): - return output_dir.joinpath(*nav_parts, "index.mdx") - return output_dir.joinpath(*nav_parts[:-1], f"{nav_parts[-1]}.mdx") + return output_dir.joinpath(_package_slug(module.package), *nav_parts, "index.mdx") + return output_dir.joinpath(_package_slug(module.package), *nav_parts[:-1], f"{nav_parts[-1]}.mdx") def _sibling_sort_key(module: ModuleDoc) -> tuple[int, str]: @@ -310,7 +354,11 @@ def _sibling_positions(modules: list[ModuleDoc]) -> dict[str, int]: by_parent: dict[tuple[str, ...], list[ModuleDoc]] = {} for module in modules: nav_parts = _nav_parts(module) - parent = tuple(nav_parts[:-1]) if len(_module_parts(module)) > 1 else () + parent = ( + (_package_slug(module.package), *nav_parts[:-1]) + if len(_module_parts(module)) > 1 + else (_package_slug(module.package),) + ) by_parent.setdefault(parent, []).append(module) positions: dict[str, int] = {} @@ -321,35 +369,20 @@ def _sibling_positions(modules: list[ModuleDoc]) -> dict[str, int]: return positions -def _write_index(output_dir: Path, modules: list[ModuleDoc]) -> None: +def _write_index(output_dir: Path) -> None: lines = [ - frontmatter("Python Library Reference", "Generated Python API reference for the nemo_relay package.", 1), - "Generated from the local `python/nemo_relay` package source.\n\n", - "## Modules\n\n", + frontmatter("Python Library Reference", "Generated Python API reference for NeMo Relay packages.", 1), + "## Packages\n\n", ] - - by_parent: dict[tuple[str, ...], list[ModuleDoc]] = {} - for module in modules: - nav_parts = _nav_parts(module) - parent = tuple(nav_parts[:-1]) if len(_module_parts(module)) > 1 else () - by_parent.setdefault(parent, []).append(module) - - def write_group(parent: tuple[str, ...], depth: int) -> None: - for module in sorted(by_parent.get(parent, []), key=_sibling_sort_key): - indent = " " * depth - lines.append(f"{indent}- [`{_leaf_title(module)}`]({_module_url(module)})") - if module.summary: - lines.append(f": {module.summary}") - lines.append("\n") - write_group(tuple(_nav_parts(module)), depth + 1) - - write_group((), 0) + for package in PACKAGES: + lines.append(f"- [`{package.distribution_name}`]({BASE_URL}/{_package_slug(package)}): {package.description}\n") (output_dir / "index.mdx").write_text("".join(lines), encoding="utf-8") def _write_module(output_dir: Path, module: ModuleDoc, position: int, modules: list[ModuleDoc]) -> None: + sidebar_title = module.package.distribution_name if len(_module_parts(module)) == 1 else _leaf_title(module) lines = [ - frontmatter(module.title, module.summary, position, sidebar_title=_leaf_title(module)), + frontmatter(module.title, module.summary, position, sidebar_title=sidebar_title), f"Generated from `{display_path(module.source)}`.\n\n", f"Module `{module.name}`.\n\n", ] @@ -396,11 +429,16 @@ def _write_module(output_dir: Path, module: ModuleDoc, position: int, modules: l output_path.write_text("".join(lines), encoding="utf-8") -def generate(package_root: Path, output_dir: Path) -> int: - modules = _discover_modules(package_root) +def generate(repo_root: Path, output_dir: Path) -> int: + modules: list[ModuleDoc] = [] + for package in PACKAGES: + package_root = repo_root / package.source_root + if not package_root.is_dir(): + raise SystemExit(f"package root not found: {package_root}") + modules.extend(_discover_modules(package, package_root)) reset_output_dir(output_dir) positions = _sibling_positions(modules) - _write_index(output_dir, modules) + _write_index(output_dir) for module in modules: _write_module(output_dir, module, positions[module.name], modules) return len(modules) @@ -408,7 +446,6 @@ def generate(package_root: Path, output_dir: Path) -> int: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--package-root", type=Path, default=Path("python/nemo_relay")) parser.add_argument( "--output-dir", type=Path, @@ -416,12 +453,8 @@ def main() -> None: ) args = parser.parse_args() - package_root = args.package_root.resolve() output_dir = args.output_dir.resolve() - if not package_root.is_dir(): - raise SystemExit(f"package root not found: {package_root}") - - count = generate(package_root, output_dir) + count = generate(Path.cwd(), output_dir) print(f"Generated Python library reference for {count} module(s) in {output_dir}") diff --git a/scripts/docs/generate_rust_library_reference.py b/scripts/docs/generate_rust_library_reference.py index 03565570b..550a6576e 100644 --- a/scripts/docs/generate_rust_library_reference.py +++ b/scripts/docs/generate_rust_library_reference.py @@ -25,12 +25,17 @@ ("nemo-relay-adaptive", "nemo_relay_adaptive", "Adaptive runtime primitives and plugin components."), ("nemo-relay-pii-redaction", "nemo_relay_pii_redaction", "PII redaction plugin components for NeMo Relay."), ("nemo-relay-ffi", "nemo_relay_ffi", "C-compatible FFI surface for NeMo Relay."), + ("nemo-relay-types", "nemo_relay_types", "Shared serializable data model types for NeMo Relay."), + ("nemo-relay-plugin", "nemo_relay_plugin", "Rust SDK and stable ABI for native NeMo Relay plugins."), + ( + "nemo-relay-worker-proto", + "nemo_relay_worker_proto", + "Versioned gRPC protocol definitions for NeMo Relay worker plugins.", + ), + ("nemo-relay-worker", "nemo_relay_worker", "Rust SDK for NeMo Relay worker plugins."), ) BASE_URL = "/reference/api/rust-library-reference" -GENERATED_BY = ( - "Generated from `cargo doc --no-deps -p nemo-relay -p nemo-relay-adaptive " - "-p nemo-relay-pii-redaction -p nemo-relay-ffi`." -) +GENERATED_BY = "Generated from `cargo doc --no-deps " + " ".join(f"-p {crate}" for crate, *_ in CRATES) + "`." TRANSLATION_TABLE = str.maketrans( { "\xa0": " ", @@ -156,14 +161,7 @@ def _run_cargo_doc(repo_root: Path) -> None: "cargo", "doc", "--no-deps", - "-p", - "nemo-relay", - "-p", - "nemo-relay-adaptive", - "-p", - "nemo-relay-pii-redaction", - "-p", - "nemo-relay-ffi", + *(argument for crate, *_ in CRATES for argument in ("-p", crate)), ], cwd=repo_root, env=env, From 0486d88b025f69fbab58e5e40b93f90831c38b3c Mon Sep 17 00:00:00 2001 From: Will Killian <2007799+willkill07@users.noreply.github.com> Date: Wed, 8 Jul 2026 10:13:08 -0400 Subject: [PATCH 4/4] chore: add NAT reviewers for docs (#384) #### Overview Update documentation CODEOWNERS entries so NAT developers are requested alongside technical writers. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Add `@nvidia/NAT-developers` to the `docs/` and `fern/` ownership rules. - Preserve the existing technical-writer reviewer on both rules. #### Where should the reviewer start? Review `.github/CODEOWNERS`; it is the only changed file. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Closes RELAY-444 ## Summary by CodeRabbit * **Chores** * Updated documentation ownership rules for repo docs-related paths. * Added an additional owner for `docs/` and `fern/` to broaden review coverage. Authors: - Will Killian (https://github.com/willkill07) Approvers: - Eric Evans II (https://github.com/ericevans-nv) URL: https://github.com/NVIDIA/NeMo-Relay/pull/384 --- .github/CODEOWNERS | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 701cd58ea..6d612452b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,6 +7,6 @@ # Attribution files require dependency approver review due to change in version(s) ATTRIBUTIONS-*.md @nvidia/nat-dep-approvers -# Documentation changes must go to technical writers -docs/ @lvojtku -fern/ @lvojtku +# Documentation changes must go to technical writers and NAT developers. +docs/ @lvojtku @nvidia/NAT-developers +fern/ @lvojtku @nvidia/NAT-developers