Skip to content

feat: answer Slack replies and mentions, with Clef deciding when to - #19

Merged
ChristopherPHolder merged 10 commits into
mainfrom
feat/slack-responder
Oct 20, 2026
Merged

ChristopherPHolder merged 10 commits into
mainfrom
feat/slack-responder

Conversation

@ChristopherPHolder

Copy link
Copy Markdown
Collaborator

Closes #18.

The bot now listens to Slack as well as posting to it. When someone mentions it or replies under one of its posts in the office channel, it replies, reacts or stays out of it. Clef on Workers AI makes that call for thread replies, and the free banter models write the replies.

⚠️ Before merging

Merging deploys, and the new Slack secrets are required, so the Slack side has to be ready first or the bot won't start:

  • Create the Slack app (or convert the webhook app): Socket Mode on, an app-level token with connections:write, bot scopes chat:write, reactions:write, reactions:read, app_mentions:read, channels:history, users:read, and events app_mention, message.channels, reaction_added, reaction_removed. If the channel is private, also groups:history / message.groups. Leave Interactivity off. Install it. The steps are in the README under Slack app.
  • Invite the bot to the office channel.
  • Add the SLACK_BOT_TOKEN, SLACK_APP_TOKEN and SLACK_CHANNEL_ID GitHub secrets.

After merging:

  • Check that the next announcement or reminder arrives and an office.slack_posts row appears.
  • Fill in office.members.slack_user_id for the people we can.
  • Tell the team the bot reads mentions and replies in the office channel, and that their messages, office activity and names go to AI models and are stored.
  • Then switch it on: UPDATE office.responder_settings SET enabled = true, updated_at = now();
  • Later, delete the SLACK_WEBHOOK_URL secret and the old webhook.

What changed

Posting. Everything now goes through chat.postMessage with the bot token instead of the Incoming Webhook. SlackError reasons now come from Slack's error codes: ratelimited/429 is retried after Retry-After; invalid_auth, token_revoked, account_inactive, channel_not_found and not_in_channel are AccessRevoked (replacing WebhookRevoked); anything else is InvalidPayload. Retries, timeouts and Sentry severities are unchanged. Every post is recorded in slack_posts with its ts, kind and model. This part is unconditional.

Listening. SlackSocket holds a Socket Mode connection built on Effect's Socket. It acks every envelope before doing anything else, drops redeliveries, reconnects straight away on disconnect, and backs off up to 5 minutes when the connection drops or Slack won't open one. It never fails the program. SlackEvent decides what's considered: mentions, and replies in threads under the bot's own posts, both only in SLACK_CHANNEL_ID. Reactions on the bot's posts are recorded raw.

Deciding. Clef goes through effect/ai's DecisionModel with a small CloudflareDecisionModel provider, called directly on Workers AI. I pinned the response schema against a real call: Clef's confidence is a separate value from the chosen option's probability, and the threshold uses confidence. The exact request is stored with each decision.

Writing. The model fallback, timeouts and daily-limit handling moved out of HeadlineWriter into AiWriting, which HeadlineWriter (unchanged interface) and the new ReplyWriter share. Replies try OpenRouter's banter models first, with Workers AI as the fallback, because Clef shares Workers AI's free daily Neurons. The daily-limit check now also recognises Cloudflare's spent allocation (code 4006), for headlines and banter too.

Responding. Responder runs as a fourth fiber next to announce and record. Each thread keeps only its latest waiting message, and mentions wait separately. One lane handles messages one at a time, in the order threads first spoke. It reads responder_settings for every message, checks office hours, staleness and the caps, asks Clef only when it could act, then replies, reacts or ignores, and records why.

Settings in the database. office.responder_settings is a single typed row with CHECK constraints, seeded with enabled = false: clef_model, min_confidence (0.7), max_replies_per_day (20), max_reactions_per_day (30), max_replies_per_thread (3), stale_after_seconds (120), clef_thread_messages (10), writer_thread_messages (50) and reaction_emoji. While it's off, nothing is decided, posted or recorded. There are no CLEF_* environment variables.

People. A Slack author is linked to an office member by the new members.slack_user_id, then by their Slack real name against real_name. Anyone else keeps their Slack name, and the prompt tells the model not to assume they're anyone in the log.

Differences from the issue

  • Per-thread mailboxes. These are a single insertion-ordered map with one lane fiber, rather than an RcMap of sliding queues with a fiber per thread. The behaviour is the same (latest message per thread wins, mentions wait separately, threads take turns), with no per-thread fibers to time out mid-reply.
  • The mention reason. I added a mention decision reason for replies to mentions, since clef would be wrong there.
  • Reactions. They're identified as the bot's by the event's item_user rather than a slack_posts lookup, so recording them doesn't need an extra query.

Testing

  • 259 tests, run against fakes and PGlite. New suites cover SlackClient error classification, SlackHistory and the migration constraints, SlackEvent eligibility, SlackSocket against a fake WebSocket (acks before hand-off, dropped redeliveries, reconnects), CloudflareDecisionModel, ReplyDecider, ReplyWriter, and 25 Responder scenarios: mention → reply, react, ignore, low confidence, each cap, outside hours, off and settings changes, superseded, stale, mentions not pushed out, turn order, and each failure staying quiet.
  • I mutation-checked the staleness, superseding, cap shortcut, enabled flag and mention mailbox rules: each mutation breaks its test.
  • I called Clef for real through ReplyDecider. "bot, how many days was Sam in last week?" got reply at 0.95. Lunch plans between colleagues got ignore. "haha Sam basically lives there" got react, but at only 0.18 confidence, so it would be ignored at 0.7. min_confidence may want tuning once there's data.
  • Not run end to end against Slack, since that needs the new app and tokens.

@ChristopherPHolder
ChristopherPHolder merged commit e276021 into main Oct 20, 2026
4 checks passed
@ChristopherPHolder
ChristopherPHolder deleted the feat/slack-responder branch October 20, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Answer Slack replies and mentions, with Clef deciding when to

1 participant