Skip to content

Commit ef590c2

Browse files
Merge upstream/main into feat/atlas-cloud-provider
Resolve the costs.mdx provider table conflict: keep upstream's 'Model calls' wording and Nebius/Kie rows, keep the Atlas Cloud row. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: binyangzhu000-sudo <224954946+binyangzhu000-sudo@users.noreply.github.com>
2 parents 1c7f5a0 + b33ac20 commit ef590c2

7,296 files changed

Lines changed: 2220233 additions & 168183 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/add-block/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,7 @@ sourceIds.forEach((id) => delete params[id]) // subblock ids are deleted
350350
if (chosen !== undefined) params[group.canonicalId] = chosen
351351
```
352352

353-
So by the time `tools.config.params(inputs)` runs (`executor/handlers/generic/generic-handler.ts`),
353+
So by the time `tools.config.params(inputs)` runs (`executor/handlers/generic/handler.ts`),
354354
`params.uploadFile` and `params.fileRef` are **gone** and the value is under `params.file`. Reading a
355355
subblock id there yields `undefined` and silently sends no file.
356356

@@ -359,7 +359,7 @@ and the first non-empty advanced value in advanced mode, so a stale value in the
359359
never leak. `normalizeFileInput` then handles the JSON string that advanced-mode template resolution
360360
produces.
361361

362-
Note that `generic-handler` merges rather than replaces (`{ ...inputs, ...transformedParams }`), so
362+
Note that `generic/handler` merges rather than replaces (`{ ...inputs, ...transformedParams }`), so
363363
omitting a key from the returned object does not strip it from what the tool receives. Tools simply
364364
ignore params they do not declare.
365365

‎.agents/skills/add-enrichment/SKILL.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Because enrichments run on Sim's hosted keys by default, **every provider tool y
2222

2323
## Architecture (what you're plugging into)
2424

25-
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, projectFailure, mapOutput }` — `toolProvider` fills `projectFailure` with the standard HTTP projection; override it only for a provider whose failure shape is nonstandard (see `enrichments/provider-failures/`). Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
25+
- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, projectFailure, mapOutput, isVerified? }` — `toolProvider` fills `projectFailure` with the standard HTTP projection; override it only for a provider whose failure shape is nonstandard (see `enrichments/provider-failures/`). Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe.
2626
- **`enrichments/providers.ts`** — `toolProvider(...)` (typed passthrough) plus shared input helpers: `str(v)`, `normalizeDomain(v)`, `firstNonEmpty(arr)`, `splitName(fullName)`, and `projectEnrichmentProviderFailure`.
2727
- **`enrichments/run.ts`** — the server-only cascade runner. Calls `executeTool(provider.toolId, { ...params, _context: { workspaceId, userId } })`, accumulates hosted-key cost, returns the first non-empty mapped result. **You do not edit this** — it works for any registry entry.
2828
- **`enrichments/registry.ts`** — `ENRICHMENT_REGISTRY` / `ALL_ENRICHMENTS` / `getEnrichment`. Register new entries here.
@@ -37,7 +37,9 @@ For each output the enrichment produces, decide which existing tool provides it.
3737
- Its `params` accept what you can derive from table columns (read the tool's `params`).
3838
- Its `outputs` / `transformResponse` actually expose the field you need (read the real output shape — don't assume).
3939

40-
Order providers **cheapest / most-likely-to-hit first**; the cascade stops at the first non-empty result. Apollo and LinkedIn APIs are not hosted-safe (ToS) — never call them as providers.
40+
Order providers by **accuracy first, then price per hit**. Order decides which answer wins when two providers would disagree, so a cheaper but less accurate provider ahead of a better one trades quality for cost; reorder for price only with benchmark evidence. Among similarly accurate providers, put the ones that bill on a miss or charge a high price per hit after the cheaper, free-on-miss ones (check each vendor's billing docs and the tool's `hosting.pricing`); never move a more accurate provider behind a less accurate one to save cost. The cascade stops at the first non-empty, verified result. Apollo and LinkedIn APIs are not hosted-safe (ToS) — never call them as providers.
41+
42+
Prefer a vendor's own verified-only option (e.g. Prospeo `only_verified_email`) when it turns an unverified result into a free miss. Otherwise, when a provider can return an unconfirmed hit (a catch-all guess, a `risky` status), declare `isVerified(output)`: an unconfirmed hit is held while later providers look for a confirmed one, and fills the cell only if none does. `isVerified: false` marks a provider that never confirms its hits (a database record); it runs only while nothing is held.
4143

4244
## Step 2: Verify hosted-key support — chain to `/add-hosted-key` if missing
4345

@@ -136,7 +138,8 @@ export const ENRICHMENT_REGISTRY: EnrichmentRegistry = {
136138

137139
- [ ] Each output mapped to a real tool field (verified against the tool's `params`/`outputs`)
138140
- [ ] **Every provider tool has a `hosting` block — ran `/add-hosted-key` for any that didn't**
139-
- [ ] Providers ordered cheapest / most-likely-first; Apollo/LinkedIn not used
141+
- [ ] Providers ordered accuracy-first, then price per hit (billing on a miss checked against vendor docs); Apollo/LinkedIn not used
142+
- [ ] Providers that can return an unconfirmed hit declare `isVerified`
140143
- [ ] Enrichment file is client-safe (no `@/tools` import); uses `toolProvider` + shared helpers
141144
- [ ] `buildParams` returns `null` on insufficient inputs; `mapOutput` returns `null` on a miss
142145
- [ ] Registered in `enrichments/registry.ts`

‎.agents/skills/add-integration/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -657,8 +657,8 @@ File Downloads and Generated Files.
657657
| Helper | Location | Purpose |
658658
|--------|----------|---------|
659659
| `normalizeFileInput` | `@/blocks/utils` | Normalize file params in block config |
660-
| `processFilesToUserFiles` | `@/lib/uploads/utils/file-utils` | Convert raw inputs to UserFile[] |
661-
| `downloadFileFromStorage` | `@/lib/uploads/utils/file-utils.server` | Get file Buffer from UserFile |
660+
| `processFilesToUserFiles` | `@/lib/uploads/utils/file` | Convert raw inputs to UserFile[] |
661+
| `downloadFileFromStorage` | `@/lib/uploads/utils/file.server` | Get file Buffer from UserFile |
662662
| `FileToolProcessor` | `@/executor/utils/file-tool-processor` | Executor-side; stores declared file outputs (not called by tools) |
663663
| `isUserFile` | `@/lib/core/utils/user-file` | Type guard for UserFile objects |
664664
| `FileInputSchema` | `@/lib/uploads/utils/file-schemas` | Zod schema for file validation |

‎.agents/skills/add-managed-cli/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Read these live sources before editing; do not copy their current entries into t
1515
2. `apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts` — server-only recipes and recipe helpers.
1616
3. `apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts` — catalog and supply-chain invariants.
1717
4. `apps/sim/lib/execution/remote-sandbox/cli-tools-boundary.test.ts` — client/server import boundary.
18-
5. `apps/sim/lib/execution/remote-sandbox/sandbox-spec.ts` — content-addressed hash inputs.
18+
5. `apps/sim/lib/execution/remote-sandbox/spec.ts` — content-addressed hash inputs.
1919

2020
Read `resolve.ts` and `e2b.ts` only when changing provisioning mechanics. A normal catalog addition should not require UI, API, database, resolver, or provider edits; those paths derive from the registries.
2121

@@ -114,7 +114,7 @@ Never commit downloaded artifacts or credentials.
114114
bun run --cwd apps/sim test \
115115
lib/execution/remote-sandbox/cli-tools.test.ts \
116116
lib/execution/remote-sandbox/cli-tools-boundary.test.ts \
117-
lib/execution/remote-sandbox/sandbox-spec.test.ts \
117+
lib/execution/remote-sandbox/spec.test.ts \
118118
lib/execution/remote-sandbox/resolve.test.ts \
119119
'app/workspace/[workspaceId]/settings/components/sandboxes/utils.test.ts'
120120
```

‎.agents/skills/add-tools/SKILL.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,11 @@ export const {serviceName}{Action}Tool: ToolConfig<
8383
description: 'Brief description', // One sentence
8484
version: '1.0.0',
8585

86-
// OAuth config (if service uses OAuth)
86+
// Required whenever the tool takes a saved credential — browser OAuth or service account
8787
oauth: {
8888
required: true,
89-
provider: '{service}', // Must match OAuth provider ID
89+
provider: '{service}', // Must match the service id in OAUTH_PROVIDERS
90+
// credentialKind: 'service-account', // service-account-only integrations
9091
},
9192

9293
params: {
@@ -186,6 +187,8 @@ fallback, or caller-controlled `_context` authority.
186187
- `'user-or-llm'` - User provides OR LLM can compute (search queries, content, filters, most fall into this category)
187188
- `'llm-only'` - Computed by the LLM only; never shown as a user field
188189

190+
A tool with a `credential` or `oauthCredential` param must declare `oauth: { required: true, provider }`, service-account-only integrations included: without it, token resolution rejects every credential at run time, and `bun run check:tool-param-reachability` fails.
191+
189192
A required `'hidden'` param needs an `oauth` declaration or `hosting.apiKeyParam` to supply it (`bun run check:tool-param-reachability`).
190193

191194
A declared `timeout` param is an ordinary tool input — put it in the request body or URL yourself if the provider expects it; it becomes Sim's millisecond request deadline only when the tool sets `timeoutParamIsDeadline: true` (e.g. `http_request`). A `method` param on a tool with a fixed `request.method` would be sent as the HTTP verb, so the same audit rejects it.

‎.agents/skills/add-trigger/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ export function build{Service}Outputs(): Record<string, TriggerOutput> {
108108

109109
```typescript
110110
import { {Service}Icon } from '@/components/icons'
111-
import { buildTriggerSubBlocks } from '@/triggers'
111+
import { buildTriggerSubBlocks } from '@/triggers/subblocks'
112112
import { build{Service}ExtraFields, build{Service}Outputs, {service}SetupInstructions, {service}TriggerOptions } from '@/triggers/{service}/utils'
113113
import type { TriggerConfig } from '@/triggers/types'
114114

‎.agents/skills/migrate-application-operation/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Read these files completely before editing:
4747
- `apps/sim/lib/api/server/routes/internal-json-route.ts`
4848
- `apps/sim/lib/api/server/routes/v2-json-route.ts`
4949
- `apps/sim/lib/auth/internal-delegation.ts`
50-
- `apps/sim/lib/mothership/application/application-adapter.ts`
50+
- `apps/sim/lib/mothership/application/adapter.ts`
5151
- `apps/sim/lib/mothership/auth/application-delegation.ts`
5252

5353
Use the file domain only as a representative golden slice:

‎.agents/skills/validate-integration/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,7 @@ For **each tool** in `tools.access`:
217217
- Long text/content → `long-input`
218218
- True/false → `switch` (a Yes/No `dropdown` only when the tool needs a third "unset" state)
219219
- Credentials → `oauth-input` with correct `serviceId`
220+
- Every tool with a `credential`/`oauthCredential` param declares `oauth: { required: true, provider }` matching that `serviceId`, plus `credentialKind: 'service-account'` when only service accounts are supported
220221
- [ ] Dropdown `value: () => 'default'` is set for dropdowns with a sensible default
221222
- [ ] Every `short-input`, `long-input`, `code`, and selector subBlock has a `placeholder` — including
222223
password fields (`Enter your API key`). Formatted values show the shape

‎.claude-plugin/marketplace.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"name": "sim",
3+
"description": "Plugins for Sim, the open-source AI workspace.",
4+
"owner": {
5+
"name": "Sim",
6+
"url": "https://www.sim.ai"
7+
},
8+
"plugins": [
9+
{
10+
"name": "sim",
11+
"source": "./plugins/sim"
12+
}
13+
]
14+
}
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
description: How stored content (workspace files, run files, knowledge, table rows) reaches Chat's model, and the per-path policy. Read before changing a use case, route or Chat surface that returns such content.
3+
paths:
4+
- "apps/sim/lib/core/application/**"
5+
- "apps/sim/lib/mothership/content-delivery/**"
6+
- "apps/sim/lib/workspace-files/application/**"
7+
- "apps/sim/lib/knowledge/application/**"
8+
- "apps/sim/lib/table/application/**"
9+
- "apps/sim/lib/workflows/application/**"
10+
- "apps/sim/lib/mothership/chat/application/**"
11+
- "apps/sim/lib/**/application/operations.ts"
12+
- "apps/sim/lib/mothership/agent-cli/**"
13+
- "apps/sim/lib/mothership/tools/**"
14+
- "apps/sim/app/api/v2/**"
15+
---
16+
17+
# Sim Content Delivery to Chat
18+
19+
Content derived from a provenance-bearing store reaches Chat's model only after **each item** it
20+
came from is admitted by the gate of the surface it leaves through. The stores are workspace files
21+
(and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers
22+
bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output,
23+
error text that quotes content, and public links. Counts, ordering, and match locations are ordinary
24+
metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional
25+
human uploads remain ordinary input and are not scanned.
26+
27+
Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are
28+
classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and
29+
the workflow executor see byte-identical results.
30+
31+
A run Chat starts executes as Chat's principal, so `executeWorkflowCore` runs every execution under
32+
`runWorkflowOutsideContentDelivery`, where that principal reads like any other. What the run returns
33+
is gated where Chat started it: run execute/resume keep inline file bytes only for admitted files
34+
(`withholdUnadmittedInlineFiles`), and Chat's in-app run tools strip `base64` from their result.
35+
36+
## The pieces
37+
38+
- `lib/core/application/content-delivery.ts`: the primitive.
39+
- `admitContent(principal, sources)` returns per-item verdicts. `requireAdmittedContent` is the
40+
single-item form and throws `ContentRefusedError` (403).
41+
- `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure
42+
does not undo the write: audit and effects run, then the operation answers with the refusal.
43+
Settled tool results use the same deferral, but do not claim that a mutation was saved.
44+
- A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry
45+
may appear), `{ derived: value }` (only entries occurring in `value`), `'opaque'` (bytes no
46+
redaction reaches: binary, base64, archives, images, public links), or `'excerpt'` (text already
47+
cut before admission, where a secret fragment cannot be redacted). Both gates withhold an
48+
excerpt with known secret entries; raw opaque bytes may still enter the workbench with history.
49+
- `redactBeforeCutting` projects complete text before selecting a line window. The session gate
50+
creates the existing redactor from the operation's bounded evidence only when a cut needs it;
51+
ordinary byte reads do not decrypt a second time or reload history.
52+
- Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries
53+
into the registry egress projects with; the **session gate** (run_code sandbox proxy) records
54+
them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct
55+
single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's
56+
registry so the whole output is withheld and the workbench copy stays unknown. It needs a
57+
registry to latch; without one it refuses. One refused item never latches the shared registry.
58+
- `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches
59+
the model (the user's own edit preview; a derived file that carries its sources' provenance).
60+
Name why at each use.
61+
62+
## Rules
63+
64+
1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result
65+
can carry content from a store, `'none'` otherwise. Definition throws
66+
without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one.
67+
2. **A `sourced` use case accounts for what it returns**, before it loads content: call
68+
`admitContent` (with `[]` when this call returns none). Otherwise Chat gets
69+
`ContentDeliveryUnaccountedError`, and the transports turn the response into a 403.
70+
3. **Classify before loading bytes**, and exclude refused items per item. Withhold a whole result
71+
only when its items cannot be separated (one file, one render, one compile).
72+
4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.**
73+
`copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive.
74+
Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`.
75+
5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for
76+
Chat goes through the gated use case, or names `observeContentOutsideModel` and why.
77+
78+
## Policy
79+
80+
U = `unknown`, E = exact with secret entries, R/C = unrecorded or clean. Each item is decided on its
81+
own unless noted.
82+
83+
| Path | U | E | R / C |
84+
|---|---|---|---|
85+
| File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Opaque: into a chat workbench with entries imported, so output is redacted; refused without one | Delivered |
86+
| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered |
87+
| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered |
88+
| File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered |
89+
| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included |
90+
| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned |
91+
| File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
92+
| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
93+
| Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
94+
| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references |
95+
| Run file download | 403 | Redacted | Delivered |
96+
| Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) |
97+
| KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered |
98+
| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source |
99+
| Table export download | Existing unsupported Chat operation | | |
100+
| Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
101+
| Row search | Locations returned without cell values | Same | Returned |
102+
| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
103+
| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization |
104+
105+
Search pagination and aggregate totals retain their database meaning even when an item or preview is
106+
withheld. Do not scan unrelated table rows or whole document collections to conceal counts.
107+
A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path.
108+
Registry admission commits only when the complete item fits the existing capacity; a refused item
109+
must not consume capacity or permanently disable the parent registry.

0 commit comments

Comments
 (0)