FreeScout: Search conversations' text through the API

freescout.wordpress.net/AGENTS.md

@@ -14,7 +14,7 @@ Context for AI coding agents and developers working on the WordPress.org FreeSco
   - `WPOrgPluginReview` — the latest plugin review of a conversation in its sidebar, for the plugins team: the details, flags, and issues of its review emails' Review ID lines. It indexes which replies are review emails, and of what type; `php artisan wporgpluginreview:index` indexes them again. Only in the plugins team's mailbox, and, with the Teams module, only for members of the teams whose names start with "Plugin". Issue names and replies to copy are in its `Services/IssueNames.php` and `Services/FlagReplies.php`. The index keeps each review email's plugin ID too, which the API's conversation list takes as `reviewPluginId`.
   - `WPOrgSearch` — gives searches without other filters core's After filter, a year ago unless the agent picks a date, and runs core's conversation search through its `search.conversations.perform` hook in one query with `COUNT(*) OVER()`, stopped after 20 seconds on MariaDB.
   - `WPOrgSidebar` — WordPress.org panels in the conversation sidebar, loaded over AJAX from `api.wordpress.org/dotorg/freescout/`.
-  - `WPOrgSite` — tweaks for how WordPress.org runs FreeScout, rather than features; add new ones here instead of starting a module. So far: the WordPress mark, a Modules page that lists only installed modules and neither updates nor deletes them, and conversation UX (keyboard shortcuts, `/` for saved replies, a collapsible sidebar, pinned notes and headers, an alternate design), each documented in its README along with the core gotchas they work around (`#app`'s `overflow: hidden`, Summernote's caret, core's global `scrollTo()`).
+  - `WPOrgSite` — tweaks for how WordPress.org runs FreeScout, rather than features; add new ones here instead of starting a module. So far: the WordPress mark, a Modules page that lists only installed modules and neither updates nor deletes them, and conversation UX (keyboard shortcuts, `/` for saved replies, a collapsible sidebar, pinned notes and headers, an alternate design), each documented in its README along with the core gotchas they work around (`#app`'s `overflow: hidden`, Summernote's caret, core's global `scrollTo()`). It also lets the API's conversation list search threads' text (`body`, bounded to reading 15,000 threads a search, as thread bodies can't be indexed).
   - `WPOrgSSO` — logs agents in through login.wordpress.org's SAML identity provider (wp-saml-idp, in the private dotorg repository), and connects every user to a WordPress.org account. Its SAML library is committed in its `vendor/`: FreeScout doesn't install module dependencies. After changing its `composer.json`, run `composer install --no-dev` in the module and commit `vendor/` (`git add -f`: the root `.gitignore` ignores it).
   - `WPOrgWebhooks` — queues conversation events to `api.wordpress.org/dotorg/freescout/webhook.php`, which records contributor stats, and keeps the copy of conversations (`wporg_helpscout`) the plugin directory reads.
 - `tests/` — shared PHPUnit bootstrap and base `TestCase`.

freescout.wordpress.net/Modules/WPOrgSite/Providers/WPOrgSiteServiceProvider.php

@@ -17,6 +17,7 @@
 use Illuminate\View\View;
 use Modules\WPOrgSite\Http\Middleware\LockModules;
 use Modules\WPOrgSite\Services\AltConversation;
+use Modules\WPOrgSite\Services\ConversationTextSearch;
 use Modules\WPOrgSite\Services\EnvSettings;
 use Modules\WPOrgSite\Services\PinnedNote;
 use Modules\WPOrgSite\Services\UserChoice;
@@ -98,6 +99,8 @@ static function ( array $styles ): array {
 			}
 		);
 
+		\Eventy::addFilter( 'api.conversations.query', array( ConversationTextSearch::class, 'filter' ), 20, 2 );
+
 		\Eventy::addFilter(
 			'javascripts',
 			static function ( array $javascripts ): array {

freescout.wordpress.net/Modules/WPOrgSite/README.md

@@ -15,6 +15,9 @@ Adapts FreeScout to how WordPress.org runs it. Small changes to FreeScout's own
 - **Pinned header:** [a conversation's header can stay in view](#pinned-header) while the agent reads the emails below it.
 - **Alternate conversation design:** agents who choose so in their profile get [conversations laid out otherwise](#alternate-conversation-design), oldest email first, with Reply and Note at the bottom.
 - **Conversations:** [keyboard shortcuts](#keyboard-shortcuts); the status and assignee selectors pinned to the top of the window while their menu is open, further down long conversations where the toolbar has scrolled out of view, so they can be chosen there, like after pressing `s` or `a`; the subject at 18px and the threads' authors at 16px, rather than core's 23px and 18px, so more of the conversation fits; and clicking the number next to the subject copies it, without the `#`. Its tooltip says when it's copied; where the browser doesn't let the page write to the clipboard, it copies through a selected field instead. It's all in `Public/js/conversation.js`.
+- **Searching conversations' text through the API:** with the API & Webhooks module, `GET /api/conversations` takes a `body` parameter, like HelpScout's `body:` search: conversations with a published email, reply, or note containing the text, matched literally and ignoring case. `bodyThreadType` (`customer`, `message`, `note`, comma-separated), `bodySince`, and `bodyBefore` (ISO 8601) narrow which threads are read. The review tools find a plugin's review with `?mailboxId=12&status=active&tag=plugin-review&body=P0TDX123HGN&bodyThreadType=message&bodySince=2026-04-01T00:00:00Z`. Code in `Services/ConversationTextSearch.php`.
+  - **Limits:** thread bodies can't be indexed, so the module's other parameters pick the conversations first, and a search over 25,000 conversations or 15,000 threads (about 2 seconds) is refused; see the `wporgsite.conversation_text_search.max_conversations` and `…max_threads` filters. Every page runs the search again.
+  - **Errors:** a text under 3 characters, an unknown thread type, a bad date, or a search over the limits answers 400, and a failure 500, as an empty list would read as no match.
 
 ## Keyboard shortcuts
 

freescout.wordpress.net/Modules/WPOrgSite/Services/ConversationTextSearch.php

@@ -0,0 +1,300 @@
+<?php
+/**
+ * Looks up conversations by the text of their threads through the API.
+ *
+ * @package WordPressdotorg\FreeScout\WPOrgSite
+ */
+
+declare( strict_types = 1 );
+
+namespace Modules\WPOrgSite\Services;
+
+use App\Thread;
+use Carbon\Carbon;
+
+/**
+ * Adds a `body` text search to the API & Webhooks module's conversation list, like HelpScout's.
+ *
+ * Thread bodies can't be indexed, so the module's other parameters pick the conversations first, and searches over
+ * MAX_CONVERSATIONS or MAX_THREADS are refused.
+ */
+final class ConversationTextSearch {
+
+	/**
+	 * The parameter, named after HelpScout's search field.
+	 *
+	 * @var string
+	 */
+	public const PARAMETER = 'body';
+
+	/**
+	 * Thread types to read: `customer`, `message`, `note`, comma-separated; all of them by default.
+	 *
+	 * @var string
+	 */
+	public const THREAD_TYPE_PARAMETER = 'bodyThreadType';
+
+	/**
+	 * Only threads created since this date.
+	 *
+	 * @var string
+	 */
+	public const SINCE_PARAMETER = 'bodySince';
+
+	/**
+	 * Only threads created before this date.
+	 *
+	 * @var string
+	 */
+	public const BEFORE_PARAMETER = 'bodyBefore';
+
+	/**
+	 * Shorter texts match too much to be worth reading every thread for.
+	 *
+	 * @var int
+	 */
+	public const MIN_LENGTH = 3;
+
+	/**
+	 * The most threads one search reads, about 1.5 seconds of review emails.
+	 *
+	 * @var int
+	 */
+	public const MAX_THREADS = 15000;
+
+	/**
+	 * The most conversations one search covers, so one without other filters is refused before listing them all.
+	 *
+	 * @var int
+	 */
+	public const MAX_CONVERSATIONS = 25000;
+
+	/**
+	 * Conversations whose threads are counted or read per query.
+	 *
+	 * @var int
+	 */
+	private const CHUNK = 500;
+
+	/**
+	 * Thread types that can be searched, by the names the API uses.
+	 *
+	 * @var array<string, int>
+	 */
+	private const THREAD_TYPES = array(
+		'customer' => Thread::TYPE_CUSTOMER,
+		'message'  => Thread::TYPE_MESSAGE,
+		'note'     => Thread::TYPE_NOTE,
+	);
+
+	/**
+	 * Limits the conversation list query to conversations with a thread containing the text.
+	 *
+	 * @param mixed $query   Query for the conversations, with the module's other parameters applied.
+	 * @param mixed $request Request.
+	 * @return mixed
+	 */
+	public static function filter( mixed $query, mixed $request ): mixed {
+		if ( ! is_object( $query ) || ! method_exists( $query, 'getQuery' ) || ! is_object( $request ) || ! method_exists( $request, 'input' ) ) {
+			return $query;
+		}
+
+		$text = $request->input( self::PARAMETER );
+
+		if ( null === $text || ( is_string( $text ) && '' === trim( $text ) ) ) {
+			return $query;
+		}
+
+		if ( ! is_string( $text ) || mb_strlen( trim( $text ) ) < self::MIN_LENGTH ) {
+			self::refuse( 400, 'The `' . self::PARAMETER . '` text needs ' . self::MIN_LENGTH . ' characters or more.' );
+		}
+
+		$threads = array(
+			'types'  => self::thread_types( $request->input( self::THREAD_TYPE_PARAMETER ) ),
+			'since'  => self::date( self::SINCE_PARAMETER, $request->input( self::SINCE_PARAMETER ) ),
+			'before' => self::date( self::BEFORE_PARAMETER, $request->input( self::BEFORE_PARAMETER ) ),
+		);
+
+		$max_conversations = (int) \Eventy::filter( 'wporgsite.conversation_text_search.max_conversations', self::MAX_CONVERSATIONS );
+		$max_threads       = (int) \Eventy::filter( 'wporgsite.conversation_text_search.max_threads', self::MAX_THREADS );
+
+		try {
+			$candidates = self::candidate_ids( $query, $max_conversations );
+			$to_read    = count( $candidates ) > $max_conversations ? 0 : self::count_threads( $query, $candidates, $threads );
+		} catch ( \Throwable $e ) {
+			\Log::error( 'WPOrgSite: conversation text search: ' . $e->getMessage() );
+			self::refuse( 500, 'The conversations could not be searched.' );
+		}
+
+		if ( count( $candidates ) > $max_conversations ) {
+			self::refuse( 400, 'The `' . self::PARAMETER . '` search reads the threads of ' . $max_conversations . ' conversations at most, and these filters match more. Narrow them with status, tag, createdSince, customerId or customerEmail.' );
+		}
+
+		if ( $to_read > $max_threads ) {
+			self::refuse( 400, 'The `' . self::PARAMETER . '` search would read ' . $to_read . ' threads, and it reads ' . $max_threads . ' at most. Narrow it with ' . self::THREAD_TYPE_PARAMETER . ', ' . self::SINCE_PARAMETER . ' and ' . self::BEFORE_PARAMETER . ', or the conversations with status, tag, createdSince, customerId or customerEmail.' );
+		}
+
+		try {
+			$matches = $to_read > 0 ? self::matching_ids( $query, $candidates, trim( $text ), $threads ) : array();
+		} catch ( \Throwable $e ) {
+			\Log::error( 'WPOrgSite: conversation text search: ' . $e->getMessage() );
+			self::refuse( 500, 'The conversations could not be searched.' );
+		}
+
+		if ( empty( $matches ) ) {
+			$query->whereRaw( '1 = 0' );
+		} else {
+			// IDs, not a correlated EXISTS, which MariaDB can turn into a scan of every thread.
+			$query->whereIn( 'conversations.id', $matches );
+		}
+
+		return $query;
+	}
+
+	/**
+	 * The thread types of the parameter, every searchable one when it names none.
+	 *
+	 * @param mixed $value Value of the parameter.
+	 * @return int[]
+	 */
+	private static function thread_types( mixed $value ): array {
+		if ( null === $value || '' === $value ) {
+			return array_values( self::THREAD_TYPES );
+		}
+
+		$types = array();
+		foreach ( explode( ',', is_string( $value ) ? $value : '' ) as $name ) {
+			$name = strtolower( trim( $name ) );
+			if ( ! isset( self::THREAD_TYPES[ $name ] ) ) {
+				self::refuse( 400, 'The `' . self::THREAD_TYPE_PARAMETER . '` parameter takes customer, message, or note, separated by commas.' );
+			}
+			$types[] = self::THREAD_TYPES[ $name ];
+		}
+
+		return array_values( array_unique( $types ) );
+	}
+
+	/**
+	 * A date parameter in the database's time zone, like the module turns `createdSince` into, or null.
+	 *
+	 * @param string $parameter Name of the parameter, for the error.
+	 * @param mixed  $value     Value of the parameter.
+	 * @return string|null
+	 * @throws \InvalidArgumentException Caught, for a value that isn't a string.
+	 */
+	private static function date( string $parameter, mixed $value ): ?string {
+		if ( null === $value || '' === $value ) {
+			return null;
+		}
+
+		try {
+			if ( ! is_string( $value ) ) {
+				throw new \InvalidArgumentException( 'Not a string' );
+			}
+
+			return Carbon::parse( $value )->setTimezone( (string) config( 'app.timezone' ) )->format( 'Y-m-d H:i:s' );
+		} catch ( \Throwable $e ) {
+			self::refuse( 400, 'The `' . $parameter . '` parameter takes a date, like 2026-04-01T00:00:00Z.' );
+		}
+	}
+
+	/**
+	 * IDs of the conversations the module's other parameters select, one more than the limit at most.
+	 *
+	 * @param mixed $query Query for the conversations.
+	 * @param int   $max   The most conversations a search covers.
+	 * @return int[]
+	 */
+	private static function candidate_ids( mixed $query, int $max ): array {
+		$candidates = ( clone $query )->getQuery();
+
+		// Without the module's columns, order, and page.
+		$candidates->columns = null;
+		$candidates->orders  = null;
+		$candidates->offset  = null;
+		$candidates->limit   = null;
+
+		$ids = $candidates->limit( $max + 1 )->pluck( 'conversations.id' )->all();
+
+		return array_values( array_unique( array_map( 'intval', $ids ) ) );
+	}
+
+	/**
+	 * The threads of the candidates a search would read, drafts included, as their state isn't indexed.
+	 *
+	 * @param mixed $query      Query for the conversations, for its connection.
+	 * @param int[] $candidates Conversation IDs.
+	 * @param array $threads    Which threads: `types`, `since`, `before`.
+	 * @return int
+	 */
+	private static function count_threads( mixed $query, array $candidates, array $threads ): int {
+		$count = 0;
+		foreach ( array_chunk( $candidates, self::CHUNK ) as $chunk ) {
+			$count += (int) self::threads_query( $query, $chunk, $threads )->count();
+		}
+
+		return $count;
+	}
+
+	/**
+	 * IDs of the conversations, among the candidates, with a published thread containing the text.
+	 *
+	 * @param mixed  $query      Query for the conversations, for its connection.
+	 * @param int[]  $candidates Conversation IDs.
+	 * @param string $text       Text to search for, matched literally.
+	 * @param array  $threads    Which threads: `types`, `since`, `before`.
+	 * @return int[]
+	 */
+	private static function matching_ids( mixed $query, array $candidates, string $text, array $threads ): array {
+		$like     = '%' . addcslashes( $text, '\\%_' ) . '%';
+		$operator = \Helper::isPgSql() ? 'ilike' : 'like';
+		$matches  = array();
+
+		foreach ( array_chunk( $candidates, self::CHUNK ) as $chunk ) {
+			$found = self::threads_query( $query, $chunk, $threads )
+				->where( 'state', Thread::STATE_PUBLISHED )
+				->where( 'body', $operator, $like )
+				->distinct()
+				->pluck( 'conversation_id' )
+				->all();
+
+			array_push( $matches, ...array_map( 'intval', $found ) );
+		}
+
+		return array_values( array_unique( $matches ) );
+	}
+
+	/**
+	 * The threads of some conversations a search covers.
+	 *
+	 * @param mixed $query   Query for the conversations, for its connection.
+	 * @param int[] $chunk   Conversation IDs.
+	 * @param array $threads Which threads: `types`, `since`, `before`.
+	 * @return mixed
+	 */
+	private static function threads_query( mixed $query, array $chunk, array $threads ): mixed {
+		$builder = $query->getQuery()->getConnection()->table( 'threads' )
+			->whereIn( 'conversation_id', $chunk )
+			->whereIn( 'type', $threads['types'] );
+
+		if ( null !== $threads['since'] ) {
+			$builder->where( 'created_at', '>=', $threads['since'] );
+		}
+		if ( null !== $threads['before'] ) {
+			$builder->where( 'created_at', '<', $threads['before'] );
+		}
+
+		return $builder;
+	}
+
+	/**
+	 * Answers with an error, as an empty list would read as no match.
+	 *
+	 * @param int    $status  HTTP status.
+	 * @param string $message Why.
+	 * @return never
+	 */
+	private static function refuse( int $status, string $message ): never {
+		abort( $status, $message );
+	}
+}

freescout.wordpress.net/Modules/WPOrgSite/module.json

@@ -3,7 +3,7 @@
     "alias": "wporgsite",
     "img": "/modules/wporgsite/img/icon.svg",
     "description": "Adapts FreeScout to how WordPress.org runs it, e.g. modules come from deploys, not from updates here.",
-    "version": "1.1.0",
+    "version": "1.2.0",
     "detailsUrl": "https://github.com/WordPress/wordpress.org/tree/trunk/freescout.wordpress.net/Modules/WPOrgSite",
     "author": "WordPress.org",
     "authorUrl": "https://wordpress.org/",