Speed up collection and map operations by nikophil · Pull Request #53 · noctud/collection

nikophil · 2026-10-05T13:54:11Z

Hello,

here is some random performance speed that I made after creating a PHPBench suite

Claude found that some of the methods were using quadratic operations (O(n²))

removeAll(), retainAll() and containsAll() look elements up by hash instead of scanning, no longer quadratic
sortBy(), sortByKey(), sortByValue() and their variants run the selector once per element and sort in C
removeEvery() no longer splices each occurrence out; indexOf() and removeFirstOccurrence() search in C
keys and values views of a map read the store arrays directly
containsValue() uses in_array()
takeFirst(), dropFirst(), takeLast() and dropLast() slice instead of walking the collection, and maps keep their hashes
Terminal operations (sum(), min(), max(), fold()…) walk the store array instead of its iterator
flip() hashes each value once
intersect(), union() and subtract() work on hashes in C, reading a set argument's hashes as they are
chunked() uses array_chunk()

you can see here the improvements, some of them are drastic, this summary only prints out the "benches" which were improved by 10% or more

just tell me if you're interested into the PHPBench suite, I'll open a PR

Both scanned the given elements with in_array() for every element of the collection, O(n * m): 50ms for 10k elements on each side. Once enough lookups are expected, StrictElementLookup indexes the elements by KeyHasher hash and confirms matches with ===, keeping the strict comparison while making each lookup O(1). Small inputs keep scanning, which is faster there.

A List (or a Map values view) answered containsAll() with one linear contains() per element, and a Sequence filtered its remaining elements with a closure on every match: both were O(n * m). They now go through StrictElementLookup, which Sequence uses to drop the elements it meets. A Set keeps its O(1) contains().

Wrapping StrictElementLookup::contains() in a closure added a method call per element, which made small removals 20-30% slower than the in_array() closure they replaced. predicate() calls in_array() directly while the elements are scanned.

The comparators called the selector on both sides of every comparison: about 2·n·log2(n) closure calls, some 20,000 for a thousand elements. The stores now select once per element and let asort() compare the selected values in C, which compares like <=> and keeps equal values in their order, as usort() did. Six to nine times faster on a thousand elements. A StringMap hands its numeric keys back as strings to the selector, as everywhere else.

removeEvery() spliced each occurrence out, shifting the rest of the list every time: quadratic when many elements matched, 1.2 ms instead of 12 µs for half of a thousand elements. It now finds them all with array_keys() and reindexes once. indexOf() and removeFirstOccurrence() find the element with array_search() instead of a PHP loop, 5 to 27 times faster. lastIndexOf() keeps its backward loop: array_keys() would scan the whole list where the loop stops at the last match.

keys->toArray() and values->toArray() went through a generator over the store's own generator: 80 µs for a thousand entries where the store already holds the arrays. The stores now hand them out directly, 15 to 200 times faster. A StringMap casts its numeric keys back to strings, as it does everywhere else.

sortByKey(), sortByValue() and their variants called the selector on both sides of every comparison, through pairs built for the occasion. They now go through the store's sortBy(), which selects once per entry and sorts in C: five to six times faster on a thousand entries.

containsValue() tested each value through a closure handed to array_any(). in_array() makes the same strict comparison in C, 27 times faster on a thousand entries.

takeFirst() and dropFirst() walked the store through a generator, and takeLast() and dropLast() handed the operation the store where it has an array_slice() path for arrays. Lists now slice their array: 13 to 50 times faster for half of a thousand elements. Maps built every [key, value] pair, walked them through a generator and hashed every key again. They now slice their store, which keeps the hashes: about 30 times faster.

The terminals shared with Sequence walked $this, that is an ArrayIterator over the store, which PHP walks about twice slower than the array itself. They now walk terminalElements(): the store array for lists and sets, $this for sequences and for the views over a map, where building every element would cost a terminal that stops early. sum(), avg(), min() and max() also tested for a selector on every element: they now pick their loop once. sum() is three times faster on a thousand elements, min() and max() about twice.

flip() checked each value's type by hand, then hashed it in containsKey() and again in put(), behind two method calls. It now hashes it once, which also rejects unsupported types, and builds the store from the hashed arrays: about twice as fast on a thousand entries.

intersect(), union() and subtract() hashed both sides in a generator, and the resulting set hashed every element again. They now key both sides by hash, read those of a hash set off its store, let array_intersect_key(), + and array_diff_key() do the work in C, and build the result from the hashed array. Two to three times faster on a thousand elements, fifty times when the other side is a set.

Collection::chunked() streamed the store element by element into a buffer. It now hands the operation the store array, which array_chunk() cuts in C: two to six times faster on a thousand elements. Sequence keeps streaming.

array_diff_key() built a whole new hash table to drop the matched indexes. With the JIT on, that made removing a single occurrence from a hundred elements 15% slower than the splice it replaced. Unsetting the indexes from a copy and reindexing it allocates no more than that splice: as fast for one occurrence, still linear for many.

codecov · 2026-10-05T13:55:51Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

takeFirst() and dropFirst() sliced the store array in each of the three collection traits, repeating what n <= 0 means. TakeOperation::first() and DropOperation::first() now slice an array themselves and stream anything else, so that sequences stay lazy.

Their tests only ran on hash maps, which left the slice() of the int and string stores untested.

Built by a map, the view holds a read-only store, so only a view built on an element store sorts a copy of that store.

delacry

Thanks for this and for the bench runs!

I've look through this, but it's a big PR.. it would take much longer to review properly so I tried my best and did several reviews with cc and inspected the findings.

reviews found two crashes and some regressions:

removeAll/retainAll/containsAll now call Hashable::identity(), which crashes on entities with a null id
containsAll() no longer stops at the first missing element (an infinite generator runs out of memory, an early miss is now ~80× slower)
set operations return wrong results after removeFirst() on int sets (needs a rebase onto #54, found it while reviewing)
takeFirst on map views, arrays in removeAll, and memory use for strings got slower

btw next time it could be split into 2 PRs like simpler improvements and then the StrictElementLookup part, so it's easier to review and there's less chance I miss something

just tell me if you're interested into the PHPBench suite, I'll open a PR

I'll plan to look into it probably more around the end of the year (around christmas) because I'll need it for some project, so I'll let you know by then, thanks!

delacry · 2026-10-09T21:59:37Z

+	 */
+	private static function hash(mixed $element): int|string
+	{
+		return is_array($element) ? 'a' : KeyHasher::hashSetKey($element);


now it calls Hashable::identity(), which is user code that can crash once it passes the indexing threshold, like the result of the method depends on element count, few pass and a lot crashes it because it runs user code differently, I think === on objects is enough, or isn't instance enough as a key?

delacry · 2026-10-09T22:06:17Z

+		}
+
+		// contains() scans the whole store here: look the elements up in it once instead.
+		$wanted = $elements instanceof Traversable ? iterator_to_array($elements, false) : array_values($elements);


iterator_to_array($elements) removes the early exit, listOf([1, 2])->containsAll($gen) used to return false at the first missing element, with the infinite generator it now runs out of memory

claude did some tests and early false is also slower, it says 0.48 ms -> 40 ms, because the whole list is indexed first, could you iterate $elements lazily? see StrictElementLookup.php:53

delacry · 2026-10-09T22:09:37Z

+		$this->elements = $elements instanceof Traversable ? iterator_to_array($elements, false) : array_values($elements);
+
+		$this->count = $count = count($this->elements);
+		if ($count * $expectedLookups > self::IndexingFactor * ($count + $expectedLookups)) {


suggested fix for CollectionLogic.php:160, which also simplifies this class (cc came with that so verify): build the index lazily inside contains()/remove(), once about IndexingFactor lookups have been scanned, instead of in the constructor from $expectedLookups. containsAll() then gets its early exit back, callers don't have to guess a lookup count, and SequenceLogic.php:411 no longer needs PHP_INT_MAX, which only works because $count * PHP_INT_MAX overflows to float.

delacry · 2026-10-09T22:12:35Z

-		$seen = [];
-		foreach ($this->data as $v) {
+		if ($data instanceof HashElementStore) {
+			return $data->toHashedArray();


i think there was found that bug I fixed in #54 so just rebase fixes that

delacry · 2026-10-09T22:19:21Z

+		if ($descending) {
+			arsort($selected);
+		} else {
+			asort($selected);


asort()/arsort() don't compare exactly like <=>.. it has something to do how php compares enums

listOf([3.0, NAN, 1.0, 2.0])->sortedByDesc(fn ($v) => $v) was [NaN, 3, 2, 1] and is now [3, NaN, 2, 1]

we can keep selector once per element and still get much faster sorting:

uasort($selected, $descending ? static fn ($a, $b) => $b <=> $a : static fn ($a, $b) => $a <=> $b);

maybe we should also add some sorrting test for this, because this is really easy to miss and if wasn't for AI reviews I had no chance to find out

delacry · 2026-10-09T22:24:56Z

 	}

+	/** @inheritDoc */
+	public function sortBy(callable $selector, bool $descending = false): void


this is repeated multiple times, maybe it should have some Operation class now, it's at IntKeyValueStore, StringKeyValueStore and AbstractElementStore, if they used shared code, it would also fixed that BC with asort and enums

delacry · 2026-10-09T22:31:29Z

+		foreach ($this->buckets[$hash] ?? [] as $candidate) {
+			if ($candidate === $element) {
+				return true;
+			}
+		}
+
+		return false;


cc found some performance regression:

All arrays share the 'a' bucket, so this loop walks them in PHP where in_array() did the same work in C: removeAll() over 3k row arrays went 80 ms → 115 ms. One line fixes it; I measured it back at base speed.

Suggested change

foreach ($this->buckets[$hash] ?? [] as $candidate) {

if ($candidate === $element) {

return true;

}

}

return false;

return in_array($element, $this->buckets[$hash] ?? [], true);

introducing a performance regression in a PR pretending to improve performance 🙃

delacry · 2026-10-09T22:35:11Z

 	public function takeFirst(int $n = 1): ImmutableCollection
 	{
-		return $this->newCollectionOf(new TakeOperation($this->store)->first($n));
+		return $this->newCollectionOf(new TakeOperation($this->store->toArray())->first($n));


toArray() on a map view builds every element just to take the first one, terminalElements already avoids this by checking instanceof

nikophil · 2026-10-10T08:28:23Z

I've look through this, but it's a big PR.. it would take much longer to review properly so I tried my best and did several reviews with cc and inspected the findings.

I forgot to say that all this PHPBench test suite was 100% pure vibe coding (hence why I did not open the PR here) - I just made it as an experiment, maybe the amount of code and the number of "benches" could (must?) be reduced

I'll dive into your review next week

nikophil added 14 commits October 5, 2026 15:53

Look map values up with in_array()

2f222de

containsValue() tested each value through a closure handed to array_any(). in_array() makes the same strict comparison in C, 27 times faster on a thousand entries.

Cut collections into chunks with array_chunk()

c65a0c5

Collection::chunked() streamed the store element by element into a buffer. It now hands the operation the store array, which array_chunk() cuts in C: two to six times faster on a thousand elements. Sequence keeps streaming.

nikophil added 3 commits October 5, 2026 16:05

Cover take and drop on int and string maps

7ea5c4e

Their tests only ran on hash maps, which left the slice() of the int and string stores untested.

Cover sorting a values view built on an element store

5a779f3

Built by a map, the view holds a read-only store, so only a view built on an element store sorts a copy of that store.

delacry requested changes Oct 9, 2026

View reviewed changes

Navigation Menu

Repository navigation

Speed up collection and map operations - #53

Speed up collection and map operations#53
nikophil wants to merge 17 commits into
noctud:0.2.xfrom
nikophil:perf/collection-operations

nikophil commented Oct 5, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 5, 2026 •

edited

Loading

Uh oh!

delacry left a comment

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

nikophil Oct 10, 2026

Uh oh!

delacry Oct 9, 2026

Uh oh!

nikophil commented Oct 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nikophil commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

delacry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikophil commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikophil commented Oct 5, 2026 •

edited

Loading

codecov Bot commented Oct 5, 2026 •

edited

Loading

nikophil commented Oct 10, 2026 •

edited

Loading