Route channel traffic into source inbox by alex-clickhouse · Pull Request #422 · ClickHouse/nerve

alex-clickhouse · 2026-09-02T16:10:27Z

Stacked on #421 (alex/slack-outbound-delivery). That one is outbound delivery; this is the channel-to-source half of Slack both directions.

What changed

Adds a per-transport channel_observations buffer and a ChannelSource for Slack and Telegram.
Configures watched traffic under slack.source.* and telegram.source.*.
Drains buffered messages into the existing source inbox as slack:observed or telegram:observed.
Keeps live routing and source routing independent.

Live route	Source matches	`include_handled_messages`	Result
accepts	no	—	channel only
accepts	yes	`false` (default)	channel only
accepts	yes	`true`	channel and source
refuses	yes	—	source only
refuses	no	—	neither

"Handled" means accepted for live routing. It does not mean the agent produced a reply.

Safety and behavior

Channel sources are disabled by default. An empty conversation allow list collects nothing.
DMs, group DMs, the agent's own messages, other bots/apps, service events, malformed events, and Telegram /pair commands are never collected.
Telegram pairing works only in a private chat with a human user. Pairing commands that fall through PTB dispatch, including media captions and commands addressed to another bot, are excluded from the source.
Telegram grants require numeric IDs. Titles, usernames, and profile names can only deny.
Telegram must receive ordinary group traffic: make the bot an admin or disable privacy mode with BotFather.
Slack IDs are recognized case-insensitively and need no API lookup. Names and globs use cached conversations.info or users.info lookups.
Drain-time deny checks use raw conversation and sender IDs. Name-based denies apply before buffering.
max_stored_messages is one trim target per transport. Trimming runs every 100 writes, so the buffer can briefly exceed the target by up to 99 rows.
Repeated buffered deliveries collapse on the inbox (source, id) key. Polling advances a cursor; daily cleanup removes expired records.

Verification

Live Slack: TestChannelSourceCollectsRealTraffic in
tests/test_slack_live_inbound.py drives the source gate from a real person
posting into a real channel, and carries one message the whole way from a
Slack event to an inbox record. Six mocked tests it subsumes were removed;
the pure branches and the conversation kinds a live test cannot provoke stay.
Full backend suite: 3810 passed, 21 skipped.
Focused routing/Slack suite: 281 passed; bootstrap/pairing suite: 120 passed.
Frontend: 114 tests passed; production build completed.
Staged diff secret scan: clean.
Manual CI for the exact PR head: run 33735810229 — backend and frontend jobs passed.

Known dependency

SourceRunner currently advances its cursor when an inbox write fails. That is tracked in #410; this PR should land after that fix or include it.

Slack already delivers every message in every channel the bot sits in; `_should_answer` throws most of them away. That traffic is exactly what a deterministic cron gate wants to fire on, so route it into the inbox where `poll_source`, `read_source`, and `MessagesGate` already live. Bridged with a spool rather than a poller. `conversations.history` would re-fetch data the socket already delivered, add latency, and need a second cursor to disagree with the first. The channel appends to `channel_observations` on the dispatch path; a `ChannelSource` drains it on the runner's cadence, which is what buys filtering, condensing, TTL, health, and cursor advance without reimplementing any of them. The spool id is AUTOINCREMENT for a load-bearing reason: the table is pruned, a plain SQLite rowid is reused once the highest row goes, and a drained-and-pruned spool would then reissue ids the cursor had passed and skip the next observations for good. Observation policy is its own gate, not the access policy. "May this person drive the agent?" and "may this room's traffic reach its inbox?" are different questions; deriving one from the other either blocks watching a channel the agent takes no orders from, or silently widens command access to everything worth watching. So `slack.observe.*` is separate from `slack.allow_channels`, and it inverts two defaults: an empty `allow_conversations` observes nothing rather than everything, and DMs are never observed — declining to answer one is a refusal, and filing it away is not what the silence led the sender to expect. Config lives with the channel because what to watch is a channel property. That leaves no `config.sync.<name>` section for `_source_schedule` to find, which returns None and silently never schedules the runner — so a runner may now carry its own schedule, and the lookup prefers it. Telegram is wired at its only seen-but-unanswered path, an unauthorized sender. That reads alarming and is the point: observation is watching people who cannot instruct the agent. What makes it safe is the explicit `allow_conversations` grant plus the inbox guardrail downstream, and that private chats are excluded. Thread parents are not expanded — `thread_ts` is spooled for a reader to follow, because fetching it would cost an API call per observation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two defects found reviewing the previous commit. `read_channel_observations` dropped unparseable rows, so the drain never learned their ids and could only advance to the last row that *parsed*. A batch of entirely unreadable rows moved the cursor nowhere, was re-read on every run, and hid everything behind it for good — the opposite of the "one bad row must not wedge the drain" the docstring claimed. It now reports every scanned row, with a None payload where the JSON failed, so the caller skips the row and still moves past it. `observe.max_spool_rows` was configured, documented, and never read: the router did not forward it, so every channel silently used the module default. Threaded through from both channel hooks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review findings. Four of these let something be observed that the stated policy says would not be. A malformed allow list granted everything it looked like it disabled. `allow_conversations: {"*": false}` survives the generic coercion as a dict, and `list()` of a dict yields its keys — so a rule that reads as a switched-off wildcard became `["*"]`. Mappings are now discarded outright. A bare scalar still wraps to one pattern, because that is how `${VAR}` arrives and the repo's own coercion invariant requires it; the first attempt at this fix dropped it and `test_config_env` caught the regression. Slack multi-person DMs were observed. They arrive as `channel_type="mpim"` on a `G` id, so testing for `D` alone filed a group DM away as if it were a channel. The check now reads the raw declared type rather than the caller's derived one — that default turns an absent type into "channel", which is exactly the ambiguity a `G` has to refuse rather than resolve. Telegram chat titles granted access. A title is set by whoever runs the group, so `allow_conversations: ["ops-room"]` admitted any group that named itself that. Titles and usernames are now `self_set_names`, which the access module already defines as deny-eligible only — the distinction existed and this code was not using it. Other bots are skipped too, matching Slack's `_is_another_app_talking`. Telegram observation now also needs `include_unauthorized_senders`, off by default. Its only seen-but-unanswered path is a sender the allowlist refused, so collecting there is a sharper edge than Slack's "in the room but not talking to me" and should be a decision rather than a side effect. `ChannelSource` is now `<channel>:observed`. Sharing the bare `telegram` name with the existing pull source meant sharing a cron job id and a cursor key: the two runners evicted each other from the scheduler and then read each other's cursor, one an integer and the other Telethon's JSON state. Also: valid JSON that is not an object (`[]`) parsed and then raised in `_to_record`, wedging the cursor behind it; an unusable `schedule` fell back to nothing, leaving the spool filling with no drain; and the row-cap counter was process-local, so a daemon restarting more often than it wrote 100 rows never enforced the cap at all. Spooled text is now capped per row, since capping row count alone bounds nothing in bytes. The docs claimed a downstream inbox guardrail that was never wired. The drain now re-applies the deny rules, which catches a conversation denied after its messages were already spooled — deny-only, because these rules match id fields and a name-based allow rule would fail closed and drop everything. The threat model is restated honestly: nothing here inspects content, and no allow/deny list separates a report from an instruction. Not fixed here, and called out in the PR: SourceRunner advances its cursor when an inbox write fails, which for observations loses them for good. That is PR #410's subject and belongs there, not duplicated in this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Both words were mine, and the repo already had better ones. "Source" is everywhere — `nerve/sources/`, `sync.*`, `source_messages`, `poll_source`, `sources: [...]` in a cron gate, the Sources page. "Observe" appeared nowhere before this branch, so the config surface asked a reader to learn a third word for a feature that already had two. It also read wrong against its own keys: `observe.schedule`, `observe.batch_size` and `observe.condense` are source-runner settings, not observation ones. So the config is `slack.source.*` / `telegram.source.*` and the class is `ChannelSourceConfig`. It stays on the channel rather than moving to `sync.slack`, which would otherwise be the obvious home and would let the `SourceRunner.schedule` machinery go away: `sync.telegram` already exists and means the Telethon pull, so observation config cannot live there, and a `telegram:observed` runner looking itself up would silently inherit the pull source's cadence. "Observed" survives where it is still the right word — `ObservedMessage`, `ObservationPolicy`, `channel_observations`, and the `slack:observed` stream name. The pair now divides the work honestly: `source` is what you enable, `observed` describes what is in it. "Spool" is a printer metaphor for a table that is really an append-only, cursor-read, TTL-retained buffer — nothing is ever dequeued from it, which is exactly what "spool" and "queue" both imply. It is prose only, so this is a rename of comments and docs plus one config key: `max_spool_rows` is now `max_stored_messages`, which says what it caps in the domain's own terms rather than the storage layer's. Nothing has shipped, so there is no compatibility shim for the old keys: an unrecognized `observe:` block is simply ignored, and the feature stays off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

"Conversations" named nothing an operator could act on. It did not say what goes in the list, and it did not match either platform's own word — the key right above it in the same block is `slack.allow_channels`. Each transport now uses its own noun: `slack.source.allow_channels` and `telegram.source.allow_chats`. Sharing one was worse than it looks — on Telegram a "channel" is a specific entity type distinct from a group, so `allow_channels` there would have named the wrong thing. `from_dict` takes the subject, so the YAML keys and every warning use the word the operator wrote; the internal fields stay transport-neutral, since ObservationPolicy is. `docs/config.md` now states per list exactly what matches, because the answer is not uniform: a Slack channel name and handle are workspace-assigned and may grant, while a Slack display name and every Telegram title and @username are picked by their owner and can only deny. One rule underneath — a name may grant only where the platform, not the subject, controls it — and it is the same rule that made the spoofable-title bug a bug. A leading `#` or `@` is now stripped. A chat client renders a channel as `#eng-backend`, so that is what gets pasted, and matching is literal: the paste silently matched nothing, which on an allow list is an inbox that stays empty with no error to explain it. Neither platform permits a name to begin with either character, so stripping is unambiguous; a `@` inside the value is untouched, so `*@example.com` still works. The same paste bug exists on the pre-existing access keys — `allow_channels: ["#eng"]` matches nothing today — but that is inbound behaviour this branch does not otherwise touch, so it is left for its own change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The observation docs were one Slack-shaped block with Telegram appended as "same keys, plus one" — which buried the transport whose behaviour differs most, in the section for the transport it differs from. Split along the division the docs already use. `config.md` is the key reference: it carries a per-source table for every existing source ("Telegram-specific", "Gmail-specific", …), so the Slack keys now live under `## Slack` and the Telegram keys under `## Telegram`, each with its own YAML example, its own key table, and its own statement of exactly what its lists match. Neither now depends on reading the other. `sources.md` is the mechanism, so the shared reasoning moved there under its own heading: why observation is a separate grant from access, why an empty allow list means nothing, why DMs are never collected, and — the part worth reading twice — precisely how little the guardrail protects against, given nothing inspects content. Both config sections link to it rather than repeating it, and the anchors are checked. Also corrected two things the earlier renames left stale: the sources.md bullet still claimed the source was named `slack` and that a gate reads `sources: [slack]`, and the guardrail bullet described an allow/deny filter when the drain re-applies deny rules only. Added a `telegram.source` block to the example config, which had a Slack one but nothing for Telegram, and a pointer from `## Sources (sync)` to the channel sections, since a reader looking for "how do I add a source" would not otherwise find them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The source was wired as a fallback for whatever the live route declined, which made its behaviour a function of the other policy rather than of its own. On Slack that silently dropped an addressed message from an unauthorized sender — refused for live handling, and never offered to a source that had asked for that channel. On Telegram the coupling was worse: its only unanswered path is an unauthorized sender, so dm_policy: open made the source permanently empty, and the population it did collect was "everyone the allowlist refused" — which needed its own opt-in (include_unauthorized_senders) to be safe, and still could not express "collect this group". Ask both questions of every eligible group message instead, then reconcile: live accepts, source does not match → channel only live accepts, source matches → channel only (default) ...with include_handled_messages → both live refuses, source matches → source only neither matches → dropped include_handled_messages replaces include_unauthorized_senders. It defaults to false so one message is not processed twice — the agent already saw it as a turn — and turning it on suits a source that is a record rather than a work queue. Authorization is no longer a second, implicit source switch; the explicit fail-closed conversation grant is the only one. Hard exclusions are unchanged: DMs and group DMs, the bot's own messages, other bots and apps, service noise, malformed events. _observe now runs on every message, so its cheap gates come first. Docs rewritten around the resulting model: "Observation" sections become "Channel source", routing is explained once with a table, the buffer cap is corrected from per-conversation to per-transport, the ID-only limit on the post-buffer deny recheck is stated, and Telegram's group-admin/privacy-mode requirement is documented — without it the bot sees no group traffic to collect at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Route Telegram commands through independent source selection, guard malformed updates, and keep pairing credentials out of the source buffer. Pairing now accepts only private human chats, including protection against anonymous-admin updates and caption fallthrough. Clarify the routing, delivery, retention, lookup, privacy-mode, and per-transport buffer behavior in the config and source docs. Treat lowercase Slack IDs as literal IDs so ID-only policies remain lookup-free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Five fixes from review. An allow/deny pattern that looks like a Slack id skips the name lookup. Matching that shape case-insensitively made ordinary names qualify: `beckyjones` is a legal handle and `buildstatus` a legal channel name, so `deny_users: [beckyjones]` stopped denying and `allow_channels: [buildstatus]` stopped granting. Only a lookup tells a lowercase id from a lowercase name, so the shape test is case-sensitive. A lowercase id still matches, because matching is case-insensitive. It costs one cached lookup. The source gate resolved the sender before it checked the conversation, so a message in an unwatched channel cost a users.info call for each speaker in every channel the bot sits in. The conversation now decides first. An upload with no comment was buffered with empty text and reached the inbox as a record holding nothing. Name the attachment from the event, and collect nothing when there is neither text nor a name. A run drained one batch. A chat busier than batch_size per tick never caught up, and the buffer's row cap then trimmed the messages the drain had not reached. A run now repeats while the source reports has_more, up to 20 batches, and each batch is durable before the next starts. max_stored_messages of 0 reads like "no limit" and told the trim to keep no rows. Values under 100 are refused. Also drop get_channel_observation_max_id, which nothing called. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The mocked tests hand _observe an event dict the test wrote, so they settle the policy branches and nothing about the events Slack sends. TestChannelSourceCollectsRealTraffic drives the same gate from a real person posting into a real channel, and checks the payload against what Slack put in the event: the conversation id, the sender, the message ts, and the channel_key built from them. Eight live tests: ordinary chatter collected without starting a turn, a conversation off the grant declined, a name grant resolved through conversations.info, a sender denied by handle through users.info, a mention answered and not also collected, include_handled_messages sending it to both, the bot's own post ignored, and one message carried the whole way from a Slack event to an inbox record. RecordingRouter grows observe() and the two waits. expect_no_observation proves the envelope reached the channel before asserting nothing came out, so a declined message is distinguishable from one Slack never delivered. live_channel now resets slack.source between tests, which it has to: a standing grant would collect the next test's traffic under this test's policy. Six mocked tests are gone, each subsumed by one of those. What stays is the pure branching and the conversation kinds a live test cannot provoke: a group DM, an ambiguous G, another app talking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

alex-clickhouse mentioned this pull request Sep 2, 2026

Give agents a tool for posting messages to Slack independently #421

Open

alex-clickhouse changed the title ~~Feed the source inbox from what a channel sees but does not answer~~ Feed the source inbox from watched channels and chats Sep 3, 2026

alex-clickhouse changed the title ~~Feed the source inbox from watched channels and chats~~ Route watched chat traffic into the source inbox Sep 3, 2026

alex-clickhouse force-pushed the alex/channel-source-bridge branch from 8b22c60 to 26cf028 Compare September 3, 2026 10:14

alex-clickhouse changed the base branch from alex/slack-channel-live-tests to alex/slack-outbound-delivery September 3, 2026 10:14

alex-clickhouse force-pushed the alex/channel-source-bridge branch 2 times, most recently from 24acb11 to 2112253 Compare September 3, 2026 12:50

alex-clickhouse changed the title ~~Route watched chat traffic into the source inbox~~ Route channel traffic into source inbox Sep 3, 2026

alex-clickhouse marked this pull request as ready for review September 3, 2026 14:06

alex-clickhouse force-pushed the alex/channel-source-bridge branch from 30c3418 to 933bc14 Compare September 3, 2026 14:37

alex-clickhouse and others added 10 commits September 3, 2026 17:12

alex-clickhouse force-pushed the alex/channel-source-bridge branch from 933bc14 to c3872b4 Compare September 4, 2026 07:13

Navigation Menu

Uh oh!

Route channel traffic into source inbox - #422

Route channel traffic into source inbox#422
alex-clickhouse wants to merge 10 commits into
alex/slack-outbound-deliveryfrom
alex/channel-source-bridge

alex-clickhouse commented Sep 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alex-clickhouse commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Safety and behavior

Verification

Known dependency

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

alex-clickhouse commented Sep 2, 2026 •

edited

Loading