Skip to content
datalayerPublic

About

☰ πŸ’¬ Datalayer Whatsapp.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Datalayer

Become a Sponsor

☰ πŸ’¬ Datalayer for WhatsApp

A business connects its own WhatsApp number to an application it deployed on Datalayer, and the application answers on that number: the same session, rules, tools, connections, approvals and record as on datalayer.ai. WhatsApp is a host of Datalayer's existing session and approval contracts (the shared host kit in agent-runtimes' src/portable/hosts, kind: "whatsapp"); it is not a separate agent runtime. The service speaks Meta's WhatsApp Business Platform (Cloud API) directly.

State: groups (src/groups.ts) and their management from the Ship card with the Official Business Account (src/manage.ts), the webhook ingress, the install with the application's own Meta app (src/install.ts) and removal (src/removal.ts), the worker's 1:1 turn (src/worker.ts, Meta behind src/meta-port.ts, WhatsApp's forms in src/render.ts), Datalayer's deliveries (src/deliveries.ts) and approvals (src/approvals.ts), the words a person writes (src/commands.ts) and the 24-hour window keeper (src/window.ts, the notice templates in templates/). Nothing is deployed.

What it does today

  • GET /whatsapp/webhook/<key> answers Meta's endpoint verification (hub.challenge) when hub.verify_token is the one derived for the key (before the number is connected too). Each installation has its own address: there is no shared one, since Datalayer has no Meta app.
  • POST /whatsapp/webhook/<key> answers 404 for a key no kept installation has, reads the raw body up to 3 MB (413 past it), checks X-Hub-Signature-256 (the HMAC-SHA256 of the raw body with that installation's own app secret; 401 for any other), reads only the entries of the installation's own business account, claims each message once by its wamid and enqueues it with its claim in one transaction, keeps each sent message's furthest status (sent, delivered, read, played, failed), hands account events to the worker with the app they came through, and answers 200 at once. A duplicate claims nothing and is answered 200; anything not persisted is answered 500, so that Meta tries again.
  • Every phone number Meta sends is dropped at the ingress: a person is their business-scoped user id (BSUID), never their number.
  • /healthz and /readyz on every role.

Connect a number

Datalayer has no Meta app for WhatsApp and is not a Meta Tech Provider. Each Datalayer application that answers on WhatsApp has its own Meta app, created by the application's developer under their own business portfolio: one app per application keeps each application's number, webhook, quality rating and billing its own. The developer creates a Meta app of type Business with the WhatsApp product, adds the number there, and makes a system user with a token for that app (permissions whatsapp_business_messaging and whatsapp_business_management, the WhatsApp business account assigned), then connects it from the application's Ship β†’ WhatsApp card in Datalayer. The card makes the pending installation at Datalayer (a one-use bind code, good for ten minutes) and posts to this service from Datalayer's web origin only (CORS).

  1. The card asks POST /whatsapp/install/webhook ({}, or {key} to show the same address again) for the app's webhook settings: {key, callback_url, verify_token, fields}. The callback is /whatsapp/webhook/<key>, a new 24-character key; the verify token is the HMAC-SHA256 of the key with the service's own verify secret (DATALAYER_WHATSAPP_VERIFY_SECRET, generated by the chart), so nothing is kept for it and Meta's verification is answered as soon as the developer saves the address; the fields are the ten this service reads (messages, the five account fields and the Groups API's four).
  2. The card posts {bind_code, webhook_key, app_id, app_secret, token, phone_number_id?, pin?, support, privacy} to POST /whatsapp/install. The service reads the token with Meta's debug_token, asked with the app's own token (<app_id>|<app_secret> as a bearer, never in an address): it must be that app's, valid, and have both permissions. The business account and the number are found from the token's whatsapp_business_management targets: the number the card said, else the only one the token reaches. The address must not be another number's.

The service then subscribes the app to the account's webhooks, registers the number for the Cloud API unless it is registered there already (platform_type CLOUD_API: registering again would need its PIN), with the PIN the developer gave or one the service makes, creates the three notice templates, binds the installation at Datalayer as <waba_id>:<phone_number_id>, registers where Datalayer pushes its deliveries, and keeps the host key, the business token, the PIN, the delivery secret and the app's secret encrypted (AES-256-GCM, each bound to its installation as <what>:<waba_id>:<phone_number_id>). A spent or expired bind code, or any refusal by Meta, keeps nothing, and a refused step is logged as install.meta_refused with the step and Meta's codes; connecting the same number again replaces its installation. The card also sends the business's support contact (said in every chat) and its https privacy page, both kept with the installation. No business token is stored in production before the threat model is signed off.

Several applications may share a business account, each with its own app: Meta sends each subscribed app the account's webhooks, a message is claimed once whichever address it came to, and an account event ends the access of the app it came through only.

Groups

Meta's Groups API lets a number with Official Business Account status create groups: invite-only (people join with the group's invite link), at most 8 participants, at most 10,000 groups a number, one Cloud API business a group; text, media and templates, no interactive message (no buttons, lists or links as buttons). Each application's groups are created by its own number, through its own app. A WhatsApp Channel (one-way broadcast) is a different product, in which an application cannot answer.

The application's Ship β†’ WhatsApp card manages them on a connected number, from Datalayer's web origin only (CORS), each call naming the installation by its uid at Datalayer (host_uid) and carrying the signed-in person's Datalayer token as a bearer: the service asks ai-agents (GET /api/ai-agents/v1/apps/hosts/<host_uid> with that token, which answers only the installation's owner) and neither keeps nor logs it.

POST Body Answer
/whatsapp/manage/official {host_uid} {status, status_message?, conditions, may_request, groups_allowed, manager_url}
/whatsapp/manage/official/request {host_uid, website, country, language?, links?} (links: 5 to 10 pages) the same, once Meta took the request (409 while one runs)
/whatsapp/manage/groups {host_uid} {groups: [{ref, group_id?, subject, state, created_at}], most_people: 8}
/whatsapp/manage/groups/create {host_uid, subject} (≀ 128 characters) the groups
/whatsapp/manage/groups/link {host_uid, group_id, reset?} {group_id, invite_link}, read from Meta each time (a reset kills every older link); never kept
/whatsapp/manage/groups/delete {host_uid, ref} the groups (deleted at Meta too: everyone is removed from it)

The status is Meta's oba_status (NOT_STARTED, PENDING, UNDER_REVIEW, APPROVED, REJECTED, EXPIRED, CANCELLED). The conditions are Meta's five, each met, not_met or unknown: the Business Messaging Policy (the business's own word), 30 days on the platform (counted here from the number's connection, a lower bound: Meta counts from its registration), the business portfolio's verification (business_verification_status), two-step verification (met when the service registered the number with its PIN, or the business gave it), the display name approved (name_status, in beta at Meta). A request goes to Meta's POST /<phone_number_id>/official_business_account; the same can be done in WhatsApp Manager (the number β†’ Profile β†’ Official business account β†’ Submit Request). Meta's refusals are said in one sentence (131215: not eligible yet).

A group is created by its subject; Meta answers its id at once or in the group_lifecycle_update webhook, matched by its request (or its subject, against Meta's list of the number's groups, when the webhook did not come). The service keeps a group's id, subject and state (creating, active, suspended, failed; migration 008), never a participant nor an invite link.

In a group, the application answers only when it is addressed: its name or the business's written in the message, the number @-mentioned, or a reply to one of the number's messages. It says nothing while the number is paused, when the group's rule is off (the channel rule deny, set on the card's Where it answers, by the group's id) or when the application is kept to direct chats (direct_only). The person asking must have linked their Datalayer account (told once a day in the group otherwise; visitors are not answered). One session per group (hostSessionUid("whatsapp", [waba_id, phone_number_id, group_id])), each turn run as the person who asked and charged to their quota, one turn at a time; a turn's place is {kind: "channel", id: <group_id>}. The answer's words and documents are written in the group; what only a 1:1 chat takes β€” buttons, lists, Open in Datalayer (a one-use link for one person), an approval's card β€” goes to the person's own chat while its window is open, with a short pointer in the group; a question's buttons too, and a press there answers in the group. An @-mention's phone number is replaced by @someone before the question reaches Datalayer. The 1:1 words (help, link, stop…) are not read in a group. Meta's group events (created, deleted, suspended, cleared) update the group; participant events are counted, their people dropped.

The app's webhook needs the four group fields besides the six of a 1:1 number: group_lifecycle_update, group_participants_update, group_settings_update, group_status_update (the card lists all ten).

Disconnect

A number stops being answered when its owner disconnects it in Datalayer (its key is then refused: found on use, and by a daily reconciliation), when the business ends the app's access to its account at Meta (account_update, for that app's numbers only), or when Meta refuses the business token (190). It is then removed once: the business token, the PIN, the app's secret and its webhook address destroyed at once, the installation ended at Datalayer, the app taken off the business account's webhooks when no other number of the account is kept through it; late messages, deliveries and presses do nothing, and every row of the number is purged 24 hours later. The number stays the business's.

A question in a chat

A linked person writes to the number; the worker:

  • reopens the 24-hour window and says first what it held for them;
  • checks their standing at Datalayer (their link), the application (the one they chose with app, else the number's default, allowed by the installation) and the quotas per person, number, organization and application, taken once per message;
  • says the disclosure line once per chat;
  • marks the message read with the typing indicator, in one call; past the indicator's 25 seconds, ⏳ on the message; past 30 seconds, one Still working on it with Stop, then at most one a minute;
  • runs the turn as the person (the chat's session for its generation, idempotent on the message, named by its id's hash: Meta's message ids may embed a phone number);
  • writes the answer whole, since WhatsApp has no edit: WhatsApp's own marks for bold, italic, strike, monospace and lists, tables of four columns or fewer in monospace, links as a numbered footer, up to 4,096 characters a message split at paragraphs into at most four messages 10 seconds apart; a longer answer, or a wider table, also comes whole as a Markdown document, with Open in Datalayer (a one-use link for the person);
  • asks the question a turn ends on as reply buttons (three short options), a list (ten) or words.

An answer holding anything shaped like a credential is never written in the chat (a message rests at Meta for 30 days): the person gets Open in Datalayer only. A turn that outlives the window ends with a notice, and the answer comes with the person's reply. A person without a link is told once a day how to link; a paused number says so once a day.

Deliveries and approvals

Datalayer pushes what happens after a turn to POST /datalayer/deliveries/{host}, signed with the installation's delivery secret (the hosts kit's route: five minutes at most, each delivery queued once by its id); the worker also reads every number's outbox, for what a push missed. A delivery reaches the person's chat through the window keeper: inside the 24-hour window at once, outside it held by reference with one notice, and shown when the person replies. A removed number's deliveries are left unacknowledged and unshown.

An approval that waits on a linked person is one card in their chat, at most 1,024 characters: the application, the action, every material value whole, its age; Approve, Reject, Show more. A press answers a second message naming the action again, Yes or No; Yes decides at Datalayer as the person, on exactly what the card showed, once. A request that changed since its card answers This changed β€” see the new request and the new card; a decided, expired or someone else's request is said so; there is no edit, so each outcome is a new message. Anything a host may not decide (today everything but mail), or that does not fit, is Review in Datalayer only, a one-use link for the person. A personal agent's operation at a company (PAP) shows its exact revision.

A personal agent's sign-in at a company is rendered as a button to the company's page through Datalayer (or the code and the page to enter it): never a token.

The words

A person writes one of these alone in a message, in any case: help (what answers here, how to reach a person at the business, and these words), status, app (a list to choose the application, for a linked Datalayer account), start over, stop, link (a one-use button to confirm on datalayer.ai), unlink, forget (unlinked, and every row naming the person deleted at once). Inside a sentence, a word is part of the question.

The 24-hour window

WhatsApp lets a business say anything for 24 hours after the person's last message; after that, only an approved template. The window keeper holds what comes later (an answer, an approval) by reference, sends at most one notice per person per 24 hours (templates/: three utility templates naming the application only, created in the business's account at install), and gives back what was held when the person replies. stop, with no answer running, ends notices to that person; a template Meta rejects or recategorizes pauses every notice of the account. The messages sent per number and month are counted against Meta's free tier of service messages.

Retention

Every hour the worker deletes chats and what they hold 30 days after the person's last message, message statuses after 30 days, claims after 8 days (Meta retries a webhook for 7), dead jobs after 14 days and quota counts after a day. Message ids are kept only as their SHA-256 (they may embed a phone number). forget deletes every row naming the person at once.

Logs and dead letters

Logs are JSON lines through the hosts kit's logger (credentials and words redacted by field and shape), and WhatsApp's own scrub on top: a phone number in any writing becomes [phone], a BSUID its pseudonym (salted with DATALAYER_WHATSAPP_LOG_SALT), wherever it appears. Spans name a number and a person by pseudonym only; counters (turns, messages by category, notices, holds, pair_rate_limited, window inside or outside) are metric lines.

A job that fails five times is dead, kept 14 days. node dist/dead-letters.js list | replay <id> | payload <id> <reason…> (in development node --import tsx src/dead-letters-cli.ts …) against DATABASE_URL, with DATALAYER_OPERATOR naming who acts: list shows metadata and hashes only, payload one dead job's for a stated reason, replay puts it back to work; every read and replay is kept with the operator's name.

Security and privacy

docs/threat-model.md (what Meta sees, what this service keeps and for how long, each threat with its control and test) and the decisions in docs/adr/: Meta's Cloud API directly (0001), the business-scoped user id as the identity with no phone number kept (0002), the 24-hour window and templates without content (0003). Drafts until signed off: no business token is stored before.

Settings

Setting What it is
NODE_ENV development, test or production
PORT The HTTP port (3000)
DATALAYER_WHATSAPP_ROLE ingress, worker, or all (one process, development only)
DATABASE_URL PostgreSQL; required in production and for every role but all
DATALAYER_WHATSAPP_VERIFY_SECRET What each webhook address's verify token is derived from, at least 32 characters
DATALAYER_WHATSAPP_LOG_SALT What log pseudonyms are salted with (optional; datalayer-whatsapp otherwise)
DATALAYER_WHATSAPP_GRAPH_URL The Graph API and its version (https://graph.facebook.com/v23.0)
DATALAYER_WHATSAPP_SECRET_KEY 32 bytes, base64: encrypts every secret at rest (host keys, business tokens)
DATALAYER_WHATSAPP_SECRET_KEY_VERSION The key's version (1)
DATALAYER_WHATSAPP_APP_URL The service's public address (the install, Datalayer's deliveries)
DATALAYER_API_URL Datalayer's API (ai-agents, the session API)
DATALAYER_WEB_URL Datalayer's web origin (links, Open in Datalayer, the Ship card)
DATALAYER_DELIVERY_POLL_SECONDS How often the outbox is read (30; 0 never)
DATALAYER_WHATSAPP_WORKERS Jobs the worker works at once on PostgreSQL (4), so stop reaches a running turn

No setting names a Meta app: each application's own app's id and secret come from its card and are kept encrypted with its installation. Keep every value in a password manager until it reaches .env (development) or the Secret; never paste one into an issue, a chat, a commit or the shell's history.

Development

npm install
npm run fetch:agent-runtimes   # standalone clone only: agent-runtimes' lib/portable, pinned in package.json
npm run dev                    # all roles in one process, in memory
npm test                       # memory; PostgreSQL too with DATALAYER_WHATSAPP_TEST_DATABASE_URL
npm run check                  # format, lint, types, the bundle gate, tests, the bundle

The PostgreSQL tests drop and remake the public schema of the database DATALAYER_WHATSAPP_TEST_DATABASE_URL names: point it at a throwaway database, never a shared one.

docker run --rm -d --name whatsapp-test-pg -e POSTGRES_PASSWORD=test -e POSTGRES_DB=whatsapp -p 55432:5432 postgres:16.4-alpine
DATALAYER_WHATSAPP_TEST_DATABASE_URL=postgres://postgres:test@127.0.0.1:55432/whatsapp npm test
docker rm -f whatsapp-test-pg

Database

npm run migrate (node dist/migrate.js in the image) applies the host kit's shared tables under the whatsapp_ prefix (claims, jobs, usage, payload access, replays), then migrations/ in order; each file is idempotent and re-applied every time. Migration 007 refuses to run while a number connected through Datalayer's former Embedded Signup is still connected (disconnect it in Datalayer; the daily reconciliation removes it), purges the removed ones, and drops their way.

Image

make image builds from a clean export of HEAD (git archive), never the working tree; make push pushes and inspects the manifest. Bump VERSION in the Makefile for every push; a pushed tag is never overwritten.

About

☰ πŸ’¬ Datalayer Whatsapp.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages