Record a flow in Chrome, narrated, straight into Intelligence with the OpenBot Recorder

[davidmckayv]

What this changes

Adds the OpenBot Recorder, a Manifest V3 extension for Chrome and Edge that captures a task from any site. A person clicks Record, does the task in their own browser while talking it through, and clicks Stop. The recording goes straight from the browser into CopilotKit Intelligence as a Product Trajectory, in that person's own Learning space, where Learning can turn it into a Skill.

A recording is the person's steps, not the page's traffic:

• Each action is one step: a click, a change (with its masked value), a select, a submit, an Enter, or a top-frame page change.
• Each step carries what's needed to find that element again: role and name, visible text, input type, a ranked list of stable selectors, frame path, page URL and title, and the element's box with the viewport size. A click on a container resolves to the button or link it was meant for, and automatic redirect chains collapse into one page change.
• The person's narration is transcribed by the browser's built-in speech recognition. Each phrase becomes a transcript event with its start time, and no audio leaves the extension or is stored.
• No standalone network events. A step carries at most the one request it sent when it saved something: a write the step caused that carries the values just typed, names a write-style operation, or is a GraphQL mutation. Telemetry, polling, presence pings, read receipts and the page's own reads are never recorded.
• Masking happens in the browser, before anything leaves. It covers passwords, card numbers, and one-time and 2FA codes (including code fields with no autocomplete hint, and codes typed one digit per box). It also covers credential headers, tokens in URLs and bodies, and spoken card numbers.

OpenBot receives nothing that was recorded. It does three things:

• Connects a browser. Settings > Browsers > Connect this browser exchanges a one-time code for an extension-only token. The token is stored hashed, works only on the Recorder's routes, and can be revoked from the same list. OpenBot Desktop can connect the Recorder on its own computer through a native messaging host.
• Hands out the capture pass. The runtime's trajectory/<id>/connect answers only the Recorder's bearer.
• Puts the trajectory in the person's own Learning space. OpenBot binds the trajectory to its person and returns that space (user- plus a hash of the OpenBot user id) to the runtime's Learning selector. It creates the space when the browser connects, and again on a capture pass if it is still missing.

Learning, per person, shared by choice:

• Every Bot a person uses draws Skills from three places: its own Skills, the person's own, and any Skills others have shared. On a name clash the Bot's own Skill wins, then the person's own. Shared Skills from two people stay distinct.
• Settings > Learning offers three choices: only my Bots (the default), everyone's Bots, or pick individual Skills to share. Each change is audited.
• A missing or slow personal space never fails a run.

Permissions are asked at install, so the first Record records. Website access is a required host permission, and a fresh install opens one setup tab that asks for the microphone. The install dialog shows the same warnings the debugger permission already implies.

Also in this branch:

• @copilotkit/runtime moves to 1.79.1 (AG-UI 1.0.2), for server-side Learning space assignment.
• Migration 0057 adds the Recorder and Learning sharing tables.
• The Bot's cloud browser and the extension share one copy of the labelling and masking rules, in shared/.

Intelligence needs settings only, no code:

• a licence that includes trajectories;
• FF_LEARNING_TRAJECTORIES on;
• an INTELLIGENCE_GATEWAY_WS_URL that people's browsers can reach;
• a credential that can create Learning containers: INTELLIGENCE_SERVICE_TOKEN with INTELLIGENCE_PROJECT_ID on a local evaluation install, or a CopilotKit CLI login on managed Intelligence. The runtime key alone cannot create a person's space.

Where it runs

• New state that outlives a request? Postgres, in migration 0057:

  • recorder_connections: hashed bearer, label, revoked flag.
  • recorder_connect_codes: hashed one-time codes, deleted on use or expiry.
  • recorder_trajectory_bindings: trajectory to person.
  • learning_person_spaces: each person's Learning space.
  • learning_skill_sharing: each person's sharing mode.
  • learning_open_skills: the individual Skills a person chose to share.

  Four per-process caches have TTLs and hold nothing that must be shared:

  • the organization sign-in verdict;
  • the Learning-space check;
  • refusal-audit de-duplication;
  • the list of shared Skills, refreshed at once on the replica that saved a change and within 30 s elsewhere.

• What happens on the second replica?

  • A connect code minted on one replica is exchanged on another through Postgres.
  • Any replica checks a bearer against its hashed row.
  • Two replicas creating the same personal space both succeed: Intelligence answers "already exists" to the second.
  • A capture pass for the same trajectory binds once: another person gets a 404.
  • A sharing change reaches other replicas within 30 s.

• Anything serialised?

  • Code exchange is a delete inside a transaction, so a code is used once.
  • Binding is an insert on the trajectory_id primary key with ON CONFLICT DO NOTHING, followed by an owner check.
  • Container creation relies on Intelligence's unique container id.
  • The cap of 10 live connect codes per person is a count-then-insert. It is a spam bound, not a security boundary: concurrent requests can exceed it by the number in flight.

• Anything fanned out to a browser? No. The extension streams to Intelligence's realtime gateway directly.

• New listener, port, or schedule? No new port and no schedule.

  • The routes are under /api/recorder/* and /api/settings/learning, behind the same ingress as the API.
  • The native messaging host is a Desktop process started by the browser on the person's own computer.

Boundary and audit

• Every acting call still goes through the gateway. The Recorder bearer cannot run a Bot or read a conversation. It reaches only the Recorder routes and the trajectory connect route, and the gate resolves the person before the pass is minted.
• New refusals and failures each write a row:
  • recorder.connected, recorder.revoked and recorder.trajectory_refused;
  • learning.sharing_changed, learning.skill_opened and learning.skill_closed.
• Nothing new is trusted from the client that the server can resolve itself.
  • The person comes from the bearer.
  • The Learning space is derived on the server from the person.
  • The extension trusts only the origin Chrome reports for externally_connectable messages.

Changelog

• CHANGELOG.md under Unreleased. It covers:
  • the Recorder;
  • Settings > Browsers and Settings > Learning;
  • the capture pass and its audit events;
  • migration 0057;
  • OPENBOT_RECORDER_EXTRA_EXTENSION_IDS (a bad value stops the server at boot);
  • the runtime 1.79.1 move to Chat Completions for openai/* models behind a non-OpenAI OPENAI_BASE_URL (COPILOTKIT_OPENAI_API=responses keeps the Responses API).

Proof

Run against a real local Intelligence (v0.13.14), in real Chrome 155 and real Edge 155, on fixture sites and real sites, including a person's own narrated sessions in a real SaaS app.

• Steps, not traffic. Replayed through the new rules, two real sessions shrink from 693 and 525 events to 55 or fewer each, with 0 network events. They read as what the person did: sign in, enter the 2FA code, open contacts, open a contact, ask the assistant (its sendMessage attached), close banners (each "mark as read" attached).
• Real Chrome. On a fixture full of polls, beacons, trace exports, prefetches and iframe traffic, the trajectory held only steps, with each form's save attached to its submit or Enter.
• Masking. I searched full dumps of Intelligence's and OpenBot's databases for every test secret, in raw, spaced and form-encoded forms: passwords, card numbers, tokens and 2FA codes (single-field and one digit per box). Every secret had 0 hits, on both capture paths. On the build before the 2FA fixes, the same runs leaked the code in 2 to 3 events.
• Voice. Narration came back as transcript events between the steps, and a spoken card number was stored as [redacted], including when the recognizer dropped digits.
• Learning. A recording joined the person's own space (user-...), active, in cpki.trajectory_container_memberships. Tests prove the Skill merge, clash rules, sharing modes and fail-soft, because Learning delivery does not run locally.
• Lifecycle:
  • reconnecting mid-recording keeps it going;
  • Disconnect ends it failed:disconnected;
  • a revoke in Settings > Browsers stops it within 30 s;
  • a browser restart sends what was kept;
  • a fresh install in Chrome and Edge opens one setup tab, Connect needs no prompt, and the first Record records.
• Checks:
  • format:check, lint, typecheck, build, drizzle-kit check and drift: all pass.
  • test:ci: 8,420 tests, 0 failures.
  • Desktop: cargo fmt, clippy -D warnings and the lib and native host tests pass.

Known limits:

• A click inside a sandboxed iframe the extension cannot enter is not recorded.
• Voice in Edge with a real microphone is unverified. With a fake microphone, Edge's built-in recognition returned empty phrases, and the popup says so.