Apply Query Loop post exclusions in PHP instead of SQL

[roborourke]

The problem

WP_Query builds its post-queries cache key from md5( serialize( $args ) . $sql ) — the query vars and the generated SQL. Two things on main make that key vary per URL:

1. The plugin puts query_id into the query vars for every core/post-template query. query_id is derived from the post ID, so every query loop the plugin touches gets a private cache entry on every URL it renders on — whether or not any of the plugin's features are enabled.
2. Excluding posts with post__not_in puts the excluded IDs into the SQL, so a loop excluding the post being viewed can never share a cached result.

Benchmarking says (1) is by far the bigger effect.

What it measures out at

Against b9f3925, on WP 6.9 / PHP 8.4 / MariaDB, 601 posts, object cache persisting between renders (what a persistent object cache does between requests):

A single query loop with no plugin settings at all, across 100 URLs:

	before	after
Database queries	306	108
Query-loop SELECTs executed	201	3
Render time	4.38 ms/URL	3.34 ms/URL

Nothing in the over-fetching machinery is doing anything here. Three distinct queries were being re-executed 201 times purely because their cache keys differed by post ID.

A 12-loop magazine page (two loops split across multiple post templates), across 40 URLs:

	before	after
Database queries	1263	72
Query-loop SELECTs executed	1201	23
Render time	50.4 ms/URL	34.5 ms/URL
Posts rendered	2720	2720 — identical, post for post

Cost, where no cache benefit is available — one cold render against an empty cache, the worst case for over-fetching:

	before	after
Database queries	87	66
Render time	74.9 ms	73.8 ms

Within noise over 20 interleaved runs each.

The admin editor is unaffected. Medians of 15 requests, two rounds: block editor 218–235 ms before / 215–221 ms after; posts list 61–65 / 62–63; site editor 117–125 / 121–125; REST wp/v2/posts 28–30 / 28. Every gap is smaller than the spread between two rounds of the same build. That is what the code predicts — the built editor bundle is byte-identical between the two, and query_loop_block_query_vars does not fire in admin.

Method and full numbers are in docs/query-caching.md.

The approach

To show 5 posts excluding the current one, fetch 6 with no exclusion at all, drop the current post in PHP, render the first 5. The query — and the cache key — is then identical on every URL.

This works because of where core writes to the cache: in WP_Query::get_posts() the object cache write happens before posts_results and the_posts (WP 6.9: ~line 3455 vs ~3633). The unfiltered superset is what gets cached; anything removed in the_posts is removed per request.

Over-fetching by exactly count( $exclude ) guarantees a full page: at most one fetched row can be dropped per excluded ID. The fetch size depends on how many IDs are excluded, not which, so it stays stable across URLs.

Scoped to non-inherited loops. Inherited loops run against the main query, whose key is per-URL regardless.

Changes

Nothing the plugin tracks reaches the cache key. generate_cache_key() strips exactly seven query vars and serialises everything else — there is no allow-list. query_id and hm_query_loop_collect_ids now travel in one var that is stripped on pre_get_posts, before the key is generated, and bound to the WP_Query instance instead. paged is only set where offset is absent, since offset overrides it in the LIMIT clause.

Deferred exclusions (inc/deferred-exclusions.php). Collects the IDs a loop wants to exclude, plans an over-fetch, drops them on the_posts at priority 9 — before post tracking at 10, so only posts that really render are recorded.

Post templates share one query. Each used to get a narrowed query of its own — smaller posts_per_page, preceding templates' IDs in post__not_in. They now all issue the loop's own unmodified query and window the results in PHP.

found_posts is corrected per request, not through the found_posts filter — that filter only runs on a cache miss and its result is baked into the shared entry.

A behaviour change worth flagging

excludeCurrent is not in WordPress 6.9 — only in core trunk. On 6.9 and earlier core ignores the attribute entirely, so this plugin does not "take it over" there; it is what makes the setting do anything. A loop whose block attributes carry excludeCurrent starts excluding the current post where the attribute previously did nothing. The benchmark shows this changing the rendered posts on 5 of 25 URLs — exactly those where the current post fell inside the window. Intended behaviour for the setting, but a behaviour change, and the docs now say so.

Falling back

Exclusion returns to SQL when the fetch would exceed hm_query_loop_max_deferred_fetch (default 100 — deep pagination fetches per_page * page + n), when hm_query_loop_defer_exclusions is filtered false, or when the query cannot reach the_posts (fields => 'ids' and suppress_filters both skip it in core).

Testing

npm run test:php — 27 assertions on the planner, running on plain PHP with WordPress stubbed, added as a CI job. Playwright: 22/22 green. PHPCS (added to main since this branch opened) passes.

Merged main in: the sticky-posts feature landed in the same places. Both compose — sticky ordering is applied in SQL, so the over-fetched set arrives already ordered and the PHP filtering and windowing preserve it. StickyPosts\QUERY_VAR does reach WP_Query, but it only repeats IDs the ORDER BY already puts in the SQL, so it adds no fragmentation of its own.

Still worth knowing

Any post meta write, on any post, bumps posts last_changed and invalidates every cached query site-wide. On a site with view counters or similar per-request meta writes that swamps all of this. docs/query-caching.md has a snippet to measure it before investing further, plus a ranked list of what is left — inherited queries, sharing result sets between sibling loops, quantising posts_per_page.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fbx2jG1eYxRD4UzUB7gxab

[github-actions]

Playwright test results

 2 failed
 20 passed

Details

 22 tests across 7 suites
 2 minutes, 34 seconds
 4106d48

Failed tests

chromium › exclude-with-post-in.spec.js › Exclude Displayed Posts with post__in › should exclude displayed posts even when post__in is set via Advanced Query Loop
chromium › exclude-with-post-in.spec.js › Exclude Displayed Posts with post__in › should properly filter post__in array when multiple post templates used

[roborourke]

@michelhm @mattheu if you do get to this next week this is some WIP for improving the db cache hit rate using a per_page approach and filtering in PHP.

A review of this would be a good starting point and testing it out on local.

[michelhm]

@michelhm @mattheu if you do get to this next week this is some WIP for improving the db cache hit rate using a per_page approach and filtering in PHP.

A review of this would be a good starting point and testing it out on local.

I reviewed and benchmarked this against a client project (28k posts, homepage + article pages). Short version: the approach works as intended, but I couldn't see a benefit with our content. The reason seems to be outside this PR.

You mentioned this was untested, so I focused mainly on measuring it. The test suites are green on the current head (8d3afeb). 27 PHP assertions and 22 Playwright - but of course those don't tell us if the cache hit rate actually improves.

The approach looks good

Moving everything into hm_query_loop_context and removing it in pre_get_posts before generate_cache_key() looks like the right approach. Also replacing a changing ID list with an integer based on the query structure makes sense.

I instrumented build_plan() on our pages and count($exclude) is stable. For example, the homepage's deferring loop reports excl=41 on every load, so the fetch size stays the same even when the posts above it change.

Why it doesn't change anything for us

I got zero difference on this project: 74 SQL / 45 cached, 37% hit rate, the same with and without the PR across 12 cold article loads.

The reason is not this PR. Our query-filter plugin adds basically the same problem again. inc/namespace.php:67 sets $query['query_id'] = $block->context['queryId'], then reads it back in pre_get_posts to build the query-{id}- URL prefix, but never removes it.

Because queryId comes from the post ID, every loop ends up with a cache key specific to that page, regardless of what this PR does.

I tested removing it at priority 11, after query-filter's transpose, and it took us from 68 → 60 queries across 12 page loads, and 41% → 49% hit rate. The output was byte-identical and filtering was still working correctly.

This fix only works together with this PR though. Without this PR, add_query_loop_used_posts() falls back to the top-level query_id and exclude-displayed breaks. So these two changes are connected.

Bucketing the fetch - tried it, but I don't recommend it

$fetch is part of the cache key, so in theory a loop excluding 5 posts and the same loop excluding 6 would create different queries.

I tried rounding the fetch up to a whole page and measured it, but got no difference at all - 76 SQL / 68 cached / 47% hit, the same with and without it, tested twice.

The variance I initially saw (fetch values of 13, 17, 22) was actually across different article pages, not the same page over time. Those loops have a per-article post__in for related articles, so they can't share the same cache entry anyway, regardless of the fetch size.

Based on that, I don't think the extra complexity is worth it. It can read up to an extra page of rows and also breaks three tests that expect the exact fetch size (expected 6, actual 10, etc.). All the behavioural assertions beside those were still passing.

I'm parking this instead of pushing it. It could be useful if we have loops that are identical except for their exclusion count - same post type, same taxonomy, no post__in - so I'm leaving the patch here for reference in case we have this situation somewhere else.

Bucketing patch (not recommended, for reference)

--- a/inc/deferred-exclusions.php
+++ b/inc/deferred-exclusions.php
@@ -262,12 +262,35 @@ function can_filter_results( array $query ): bool {
 	return ! in_array( $query['fields'] ?? '', [ 'ids', 'id=>parent' ], true );
 }
 
+/**
+ * Granularity the over-fetch is rounded up to.
+ *
+ * A page of results is the natural unit: it is already the size the loop thinks
+ * in, and it keeps the worst-case waste to one page of rows.
+ *
+ * @param int $loop_per_page Posts the loop renders per page.
+ * @return int Bucket size, or 0 to round not at all.
+ */
+function bucket_size( int $loop_per_page ): int {
+	/**
+	 * Filters the granularity the deferred-exclusion over-fetch rounds up to.
+	 *
+	 * Larger buckets share cache entries more widely and read more rows to do it.
+	 * Return 0 to fetch exactly what the exclusions require.
+	 *
+	 * @param int $bucket        Defaults to the loop's page size.
+	 * @param int $loop_per_page Posts the loop renders per page.
+	 */
+	return max( 0, (int) apply_filters( 'hm_query_loop_fetch_bucket', $loop_per_page, $loop_per_page ) );
+}
+
 /**
  * Work out what to fetch so the loop can be assembled in PHP afterwards.
  *
  * Over-fetching by `count( $exclude )` guarantees a full page after filtering:
  * at most one fetched post can be dropped per excluded ID, so at least as many
- * survive as the loop asked for.
+ * survive as the loop asked for. The fetch is then rounded up — see
+ * bucket_size() — which only ever adds to that margin.
  *
  * @param array $query   Query vars for the loop.
  * @param int[] $exclude Post IDs to exclude.
@@ -317,6 +340,18 @@ function build_plan( array $query, array $exclude, int $page, array $context ):
 		$base_offset = $plan['fetch_offset'] - ( $loop_per_page * ( $page - 1 ) );
 		$fetch       = ( $loop_per_page * $page ) + count( $exclude );
 
+		// The fetch size lands in the cache key, so letting it track the exclusion
+		// count exactly puts the variance straight back where it was taken from —
+		// a loop excluding five posts and the same loop excluding six issue
+		// different queries. Rounding up to a whole number of pages collapses that
+		// into one query per page of results. The extra rows are trimmed in PHP,
+		// so only the size of the fetch changes, never what the loop renders.
+		$bucket = bucket_size( $loop_per_page );
+
+		if ( $bucket > 0 ) {
+			$fetch = (int) ( ceil( $fetch / $bucket ) * $bucket );
+		}
+
 		// A negative base offset means the offset was not built by core's
 		// formula, and re-slicing would silently move the window. Past
 		// get_max_fetch() the over-fetch costs more than the shared cache entry

[roborourke]

Benchmarks

WordPress 6.9 and 7.1, PHP 8.4, MariaDB same host, Twenty Twenty-Five, 601 posts. Each "URL" is a full block render with the object cache persisting between renders, as a persistent object cache does between requests. Compared against b9f3925.

A plain query loop, no plugin settings, 100 URLs

	6.9 before	6.9 after	7.1 before	7.1 after
DB queries	308	110	306	108
Loop SELECTs executed	201	3	201	3
Render time (ms/URL)	3.94	2.92	4.61	3.38

This loop has no exclusion settings, so none of the over-fetching does anything. The whole gain is that query_id no longer goes into the query vars. It is derived from the post ID, and was set for every post-template query — so three distinct queries were being re-executed 201 times purely because their cache keys differed. That affects every query loop the plugin touches, not just ones using exclusion.

12-loop page (two split across multiple post templates), 40 URLs

	6.9 before	6.9 after	7.1 before	7.1 after
DB queries	1263	72	1261	70
Loop SELECTs executed	1201	23	1201	23
Render time (ms/URL)	55.3	36.4	56.1	38.3

Cost: one cold render, empty cache (worst case for over-fetching)

	6.9 before	6.9 after	7.1 before	7.1 after
DB queries	87	66	87	66
Render time	74.9 ms	73.8 ms	77.9 ms	76.6 ms

Within noise over 15–20 interleaved runs per build.

Admin

Medians of 15 requests, two rounds, shown as a range across rounds.

	6.9 before	6.9 after	7.1 before	7.1 after
Block editor (post.php)	218–235 ms	215–221 ms	257–265 ms	251–252 ms
Posts list (edit.php)	61–65 ms	62–63 ms	72–74 ms	71–76 ms
Site editor	117–125 ms	121–125 ms	138–139 ms	131–145 ms
REST wp/v2/posts	28–30 ms	28 ms	41–43 ms	38–40 ms

Every gap is smaller than the spread between two rounds of the same build. Expected: the built editor bundle is byte-identical and query_loop_block_query_vars does not fire in admin.

Rendered output, compared post by post across 25 URLs

Fixture	6.9	7.1
12-loop page	identical	identical
Plain loop	identical	identical
Loop with excludeCurrent	differs on 5 of 25	identical

The one divergence is a core version boundary, measured rather than assumed: excludeCurrent is absent from 6.9 and 7.0, and landed in 7.1. On versions without it, core ignores the attribute, so before leaves the current post in its own "more like this" list and after removes it — the setting starting to work where it previously did nothing. On 7.1 the change is output-identical.

Full method and numbers in docs/query-caching.md.

---

Generated by Claude Code

[roborourke]

The 2 Playwright failures are pre-existing, not from this PR

Both failures are in exclude-with-post-in.spec.js and both time out on the same line — clicking Advanced Query Loop's combobox "Posts to Include", which is editor setup, before any of this plugin's behaviour is exercised.

Control experiment: I re-ran main's last green Playwright run (run 32068099228, commit b9f3925, which passed 22/22 on 17 Aug). Same commit, same workflow, fresh dependencies today:

	main @ b9f3925 (17 Aug)	main @ b9f3925 (re-run today)	this PR
Result	22 passed	20 passed, 2 failed	20 passed, 2 failed
Failing specs	—	exclude-with-post-in.spec.js:12, :193	same two
Failure	—	getByRole('combobox', { name: 'Posts to Include' }) timeout	same locator

Unchanged code, same failure. So this is drift in the test environment, not a regression here.

Why it's not this branch, independently of that: this PR ships no editor JavaScript — the built bundle is byte-identical to main's — and its PHP hooks either don't fire in admin (query_loop_block_query_vars) or bail immediately without a bound context (pre_get_posts, the_posts).

Likely cause: .wp-env.json pins core to the WordPress/WordPress#6.9 branch, not a tag, so @wordpress/components drifts within 6.9.x — a change to FormTokenField's ARIA role would break a role=combobox locator without any code change here. Advanced Query Loop is also installed unpinned from downloads.wordpress.org, though its source has only a readme commit since 17 Aug. This suite has form for this: see fix/e2e-admin-email-confirmation-timebomb.

I've not tried to fix it inside this PR, since it is unrelated to the change and would muddy the diff. Happy to take it as a separate piece of work — pinning core and AQL to fixed versions would stop the class of failure, not just this instance.

---

Generated by Claude Code