Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions .agents/skills/agent-dx-cli-scale/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
name: agent-dx-cli-scale
description: A scoring scale for evaluating how well a CLI is designed for AI agents, based on the "Rewrite Your CLI for AI Agents" principles.
---

# Agent DX CLI Scale

Use this skill to **evaluate any CLI** against the principles of agent-first design. Score each axis from 0–3, then sum for a total between 0–21.

> Human DX optimizes for discoverability and forgiveness.
> Agent DX optimizes for predictability and defense-in-depth.
> — [You Need to Rewrite Your CLI for AI Agents](/posts/rewrite-your-cli-for-ai-agents)

---

## Scoring Axes

### 1. Machine-Readable Output

Can an agent parse the CLI's output without heuristics?

| Score | Criteria |
| ----- | ----------------------------------------------------------------------------------------------------- |
| 0 | Human-only output (tables, color codes, prose). No structured format available. |
| 1 | `--output json` or equivalent exists but is incomplete or inconsistent across commands. |
| 2 | Consistent JSON output across all commands. Errors also return structured JSON. |
| 3 | NDJSON streaming for paginated results. Structured output is the default in non-TTY (piped) contexts. |

### 2. Raw Payload Input

Can an agent send the full API payload without translation through bespoke flags?

| Score | Criteria |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Only bespoke flags. No way to pass structured input. |
| 1 | Accepts `--json` or stdin JSON for some commands, but most require flags. |
| 2 | All mutating commands accept a raw JSON payload that maps directly to the underlying API schema. |
| 3 | Raw payload is first-class alongside convenience flags. The agent can use the API schema as documentation with zero translation loss. |

### 3. Schema Introspection

Can an agent discover what the CLI accepts at runtime without pre-stuffed documentation?

| Score | Criteria |
| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Only `--help` text. No machine-readable schema. |
| 1 | `--help --json` or a `describe` command for some surfaces, but incomplete. |
| 2 | Full schema introspection for all commands — params, types, required fields — as JSON. |
| 3 | Live, runtime-resolved schemas (e.g., from a discovery document) that always reflect the current API version. Includes scopes, enums, and nested types. |

### 4. Context Window Discipline

Does the CLI help agents control response size to protect their context window?

| Score | Criteria |
| ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Returns full API responses with no way to limit fields or paginate. |
| 1 | Supports `--fields` or field masks on some commands. |
| 2 | Field masks on all read commands. Pagination with `--page-all` or equivalent. |
| 3 | Streaming pagination (NDJSON per page). Explicit guidance in context/skill files on field mask usage. The CLI actively protects the agent from token waste. |

### 5. Input Hardening

Does the CLI defend against the specific ways agents fail (hallucinations, not typos)?

| Score | Criteria |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | No input validation beyond basic type checks. |
| 1 | Validates some inputs, but does not cover agent-specific hallucination patterns (path traversals, embedded query params, double encoding). |
| 2 | Rejects control characters, path traversals (`../`), percent-encoded segments (`%2e`), and embedded query params (`?`, `#`) in resource IDs. |
| 3 | Comprehensive hardening: all of the above, plus output path sandboxing to CWD, HTTP-layer percent-encoding, and an explicit security posture — _"The agent is not a trusted operator."_ |

### 6. Safety Rails

Can agents validate before acting, and are responses sanitized against prompt injection?

| Score | Criteria |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | No dry-run mode. No response sanitization. |
| 1 | `--dry-run` exists for some mutating commands. |
| 2 | `--dry-run` for all mutating commands. Agent can validate requests without side effects. |
| 3 | Dry-run plus response sanitization (e.g., via Model Armor) to defend against prompt injection embedded in API data. The full request→response loop is defended. |

### 7. Agent Knowledge Packaging

Does the CLI ship knowledge in formats agents can consume at conversation start?

| Score | Criteria |
| ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Only `--help` and a docs site. No agent-specific context files. |
| 1 | A `CONTEXT.md` or `AGENTS.md` with basic usage guidance. |
| 2 | Structured skill files (YAML frontmatter + Markdown) covering per-command or per-API-surface workflows and invariants. |
| 3 | Comprehensive skill library encoding agent-specific guardrails (_"always use --dry-run"_, _"always use --fields"_). Skills are versioned, discoverable, and follow a standard like OpenClaw. |

---

## Interpreting the Total

| Range | Rating | Description |
| ----- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| 0–5 | **Human-only** | Built for humans. Agents will struggle with parsing, hallucinate inputs, and lack safety rails. |
| 6–10 | **Agent-tolerant** | Agents can use it, but they'll waste tokens, make avoidable errors, and require heavy prompt engineering to compensate. |
| 11–15 | **Agent-ready** | Solid agent support. Structured I/O, input validation, and some introspection. A few gaps remain. |
| 16–21 | **Agent-first** | Purpose-built for agents. Full schema introspection, comprehensive input hardening, safety rails, and packaged agent knowledge. |

---

## Bonus: Multi-Surface Readiness

Not scored, but note whether the CLI exposes multiple agent surfaces from the same binary:

- [ ] **MCP (stdio JSON-RPC)** — typed tool invocation, no shell escaping
- [ ] **Extension / plugin install** — agent treats the CLI as a native capability
- [ ] **Headless auth** — env vars for tokens/credentials, no browser redirect required
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { AccountId, Hex, Node } from "enssdk";
import type { AccountId, Node } from "enssdk";

import type { ResolverRecordsSelection } from "@ensnode/ensnode-sdk";

Expand Down Expand Up @@ -50,7 +50,7 @@ function resolveOperationWithIndex(op: Operation, records: IndexedRecords): Oper
case "addr": {
const coinType = op.args[1];
const found = records?.addressRecords.find((r) => r.coinType === coinType);
return { ...op, result: (found?.value ?? null) as Hex | null };
return { ...op, result: found?.value ?? null };
Comment thread
shrugs marked this conversation as resolved.
}
case "text": {
const key = op.args[1];
Expand Down
2 changes: 1 addition & 1 deletion apps/ensapi/src/lib/resolution/execute-operations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ export function interpretOperationWithRawResult(call: Operation, raw: unknown):
case "name":
return { ...call, result: interpretNameRecordValue(asLiteralName(raw as string)) };
case "addr":
return { ...call, result: interpretAddressRecordValue(raw as string) };
return { ...call, result: interpretAddressRecordValue(raw as Hex) };
case "text":
return { ...call, result: interpretTextRecordValue(raw as string) };
case "contenthash":
Expand Down
32 changes: 16 additions & 16 deletions apps/ensapi/src/lib/resolution/make-records-response.test.ts
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
import { asInterpretedName, type CoinType, type Hex, type InterfaceId } from "enssdk";
import { asInterpretedName, type CoinType, type InterfaceId } from "enssdk";
import { describe, expect, it } from "vitest";

import type { ResolverRecordsSelection } from "@ensnode/ensnode-sdk";

import { makeRecordsResponse } from "./make-records-response";
import { makeOperations, type Operation } from "./operations";

describe("makeRecordsResponse", () => {
const node = `0x${"00".repeat(32)}` as Hex;
const ZERO_NODE = `0x${"00".repeat(32)}` as const;

describe("makeRecordsResponse", () => {
it("writes a resolved name record", () => {
const operations = [
{ functionName: "name", args: [node], result: asInterpretedName("test.eth") },
{ functionName: "name", args: [ZERO_NODE], result: asInterpretedName("test.eth") },
] satisfies Operation[];
expect(makeRecordsResponse(operations)).toEqual({ name: "test.eth" });
});

it("writes resolved address records keyed by CoinType", () => {
const operations = [
{ functionName: "addr", args: [node, 60n], result: "0x1234" as Hex },
{ functionName: "addr", args: [node, 1001n], result: "0x5678" as Hex },
{ functionName: "addr", args: [ZERO_NODE, 60n], result: "0x1234" },
{ functionName: "addr", args: [ZERO_NODE, 1001n], result: "0x5678" },
] satisfies Operation[];
expect(makeRecordsResponse(operations)).toEqual({
addresses: { 60: "0x1234", 1001: "0x5678" },
Expand All @@ -28,8 +28,8 @@ describe("makeRecordsResponse", () => {

it("writes resolved text records keyed by key", () => {
const operations = [
{ functionName: "text", args: [node, "com.twitter"], result: "@test" },
{ functionName: "text", args: [node, "avatar"], result: "ipfs://..." },
{ functionName: "text", args: [ZERO_NODE, "com.twitter"], result: "@test" },
{ functionName: "text", args: [ZERO_NODE, "avatar"], result: "ipfs://..." },
] satisfies Operation[];
expect(makeRecordsResponse(operations)).toEqual({
texts: { "com.twitter": "@test", avatar: "ipfs://..." },
Expand All @@ -38,14 +38,14 @@ describe("makeRecordsResponse", () => {

it("writes resolved contenthash / pubkey / dnszonehash / version", () => {
const operations = [
{ functionName: "contenthash", args: [node], result: "0xdeadbeef" as Hex },
{ functionName: "contenthash", args: [ZERO_NODE], result: "0xdeadbeef" },
{
functionName: "pubkey",
args: [node],
result: { x: `0x${"11".repeat(32)}` as Hex, y: `0x${"22".repeat(32)}` as Hex },
args: [ZERO_NODE],
result: { x: `0x${"11".repeat(32)}`, y: `0x${"22".repeat(32)}` },
},
{ functionName: "zonehash", args: [node], result: "0xcafe" as Hex },
{ functionName: "recordVersions", args: [node], result: 7n },
{ functionName: "zonehash", args: [ZERO_NODE], result: "0xcafe" },
{ functionName: "recordVersions", args: [ZERO_NODE], result: 7n },
] satisfies Operation[];
expect(makeRecordsResponse(operations)).toEqual({
contenthash: "0xdeadbeef",
Expand All @@ -59,8 +59,8 @@ describe("makeRecordsResponse", () => {
const operations = [
{
functionName: "ABI",
args: [node, 1n],
result: { contentType: 1n, data: "0xabcd" as Hex },
args: [ZERO_NODE, 1n],
result: { contentType: 1n, data: "0xabcd" },
},
] satisfies Operation[];
expect(makeRecordsResponse(operations)).toEqual({
Expand All @@ -82,7 +82,7 @@ describe("makeRecordsResponse", () => {
interfaces: [id],
};
// operations generated from selection, every entry unresolved
const operations = makeOperations(node, selection);
const operations = makeOperations(ZERO_NODE, selection);
expect(makeRecordsResponse(operations)).toEqual({
name: null,
addresses: { 60: null },
Expand Down
4 changes: 2 additions & 2 deletions apps/ensapi/src/omnigraph-api/schema/account-id.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ export const AccountIdRef = builder.objectRef<AccountId>("AccountId");
AccountIdRef.implement({
description: "A CAIP-10 Account ID including chainId and address.",
fields: (t) => ({
chainId: t.expose("chainId", { type: "ChainId", nullable: false }),
address: t.expose("address", { type: "Address", nullable: false }),
chainId: t.field({ type: "ChainId", nullable: false, resolve: (parent) => parent.chainId }),
address: t.field({ type: "Address", nullable: false, resolve: (parent) => parent.address }),
}),
});

Expand Down
19 changes: 9 additions & 10 deletions apps/ensapi/src/omnigraph-api/schema/records.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,18 @@ ResolvedRawTextRecordRef.implement({
description:
"A resolved 'raw' text record for an ENS name. Value is any possible string and may require additional validation or preprocessing before use.",
fields: (t) => ({
key: t.exposeString("key", {
key: t.field({
type: "String",
description: "The text record key.",
nullable: false,
resolve: (r) => r.key,
}),
value: t.exposeString("value", {
value: t.field({
type: "String",
description:
"The 'raw' text record value, or null if not set. Value is any possible string and may require additional validation or preprocessing before use.",
nullable: true,
resolve: (r) => r.value,
}),
}),
});
Expand All @@ -43,11 +47,12 @@ ResolvedAddressRecordRef.implement({
nullable: false,
resolve: (r) => r.coinType,
}),
address: t.expose("address", {
address: t.field({
type: "Hex",
description:
'The "raw" resolved address record as hex, or null if not set. Decode with ENSIP-9 (https://docs.ens.domains/ensip/9) and address-encoder (https://github.com/ensdomains/address-encoder) for the associated coin type. May be a hex value representing 0 or more bytes. There is no guarantee that an EVM coinType returns an address value of any particular byte length.',
'The "raw" resolved address record as hex, or null if not set, empty ("0x"), or zeroAddress. Decode with ENSIP-9 (https://docs.ens.domains/ensip/9) and address-encoder (https://github.com/ensdomains/address-encoder) for the associated coin type. Guaranteed to be at least one byte of hex data. There is no guarantee that an EVM CoinType returns an address value of any particular byte length.',
nullable: true,
resolve: (r) => r.address,
}),
}),
});
Expand Down Expand Up @@ -132,12 +137,6 @@ export const ResolvedRecordsRef = builder.objectRef<ResolvedRecordsModel>("Resol
ResolvedRecordsRef.implement({
description: "Records resolved for a specific ENS name via the ENS protocol.",
fields: (t) => ({
id: t.field({
description: "Stable cache key for these records: the InterpretedName used to resolve them.",
type: "InterpretedName",
nullable: false,
resolve: (parent) => parent.id,
}),
reverseName: t.string({
description:
"The `name` record value used in Reverse Resolution (ENSIP-19), or null if not set. To reduce a common point of developer confusion the Omnigraph API represents this as the `reverseName` rather than the `name` record which is what this field actually resolves to onchain.",
Expand Down
8 changes: 6 additions & 2 deletions apps/ensapi/src/omnigraph-api/schema/resolution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,17 @@ export const AccelerationStatusRef =
AccelerationStatusRef.implement({
description: "Execution status metadata for a resolver strategy.",
fields: (t) => ({
requested: t.exposeBoolean("requested", {
requested: t.field({
type: "Boolean",
description: "Whether protocol acceleration was requested by the caller.",
nullable: false,
resolve: (parent) => parent.requested,
}),
attempted: t.exposeBoolean("attempted", {
attempted: t.field({
type: "Boolean",
description: "Whether protocol acceleration was attempted at runtime.",
nullable: false,
resolve: (parent) => parent.attempted,
}),
}),
});
Expand Down
3 changes: 2 additions & 1 deletion apps/ensapi/src/omnigraph-api/schema/resolver-records.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,11 @@ ResolverRecordsRef.implement({
////////////////////////
// ResolverRecords.name
////////////////////////
name: t.expose("name", {
name: t.field({
description: "The `name` record for this `node`, if any.",
type: "String",
nullable: true,
resolve: (parent) => parent.name,
}),

////////////////////////
Expand Down
Loading
Loading