Repository navigation
neighbors cannot tell merged rows from closed ones or filter by time, so a superseded-PR check takes ~18 s per PR #238
Description
Summary
A common cleanup question on a large repository is "was this open PR already landed by a later one?". gitcrawl's neighbors is the natural tool: the answer is among the similar pull requests merged after the open PR was created. Today neighbors cannot tell a merged row from a closed one and cannot filter by time, and its rows carry no state. The only route is --include-closed with a very large --limit, plus a separate threads --include-closed --json dump to look up each row's merge time. That scores every stored vector on every call: about 18 s per PR on a warm cache, whatever --limit is, and rows that rank below the limit are lost.
How often the question comes up
On openclaw/openclaw, 2026-09-01 to 2026-10-09, external pull requests (from authors outside the maintainer team, bots excluded) closed without merging:
- 621 of 1,866 (33%) were closed with a comment saying another pull request had landed, superseded or duplicated them (classified from the closing comments; about 85% correct in a 60-row hand check).
- 437 of those name the pull request that had merged first. When that pull request's body mentioned the earlier one (129), it was closed at a median of 0.0 h after the merge; when it did not (308), the median was 26.4 h and 51% took more than a day.
The delay sits in the cases where the later pull request did not mention the earlier one, which is what a similarity search with state and time can surface.
What happens today
gitcrawl main bd30790, on a read-only copy of an openclaw/openclaw archive (165,815 threads, 3,306 open pull requests, last complete sync 2026-10-10T09:06:46Z), embeddings qwen3-embedding:0.6b (1024 dimensions, title_original basis):
gitcrawl neighbors openclaw/openclaw --number <open PR> --include-closed --limit 32765 --json
gitcrawl threads openclaw/openclaw --include-closed --json # merge times, then a local joinneighbors: 18.3 s median per PR with a warm page cache (10 open PRs; 18.0–18.5 s at--limit1,000, 5,000 or 32,765), 38–39 s cold, about 3.3 GB RSS. It decodes and scores all 165,815 stored vectors each time.- The
threadsdump (992 MB of JSON) adds 5.5 s warm or 32 s cold. --limitabove 32,765 fails withtoo many SQL variables, so a candidate ranked lower can never be returned.
Proposal
Add time filters to neighbors, applied in SQL before vectors are scored and before the limit:
--merged-after <time>: pull requests merged after that time.--open-at <time>: rows open at that time.--created-after <time|ref>: rows created after that time; a thread reference compares numbers, which GitHub assigns in creation order.
Each row would also carry state, html_url, is_draft and, when set, the author, author association, and created/closed/merged times, and the output would report state_as_of, the start of the newest complete sync. Existing fields and their order, the default query, and the schema would stay as they are.
With such a change, the same question took a median 1.4 s per PR warm (p95 4.3 s over all 3,306 open PRs) on the same copy, and returned the same top 10 as the workaround above in all 40 checks (10 PRs, two questions, warm and cold). I have this ready as a pull request.
What the candidates are worth
Similarity gives candidates, not proof. To see how often the top candidate is the answer, for every open external pull request on the same copy (2,089) I took the most similar pull request by a different author merged after it was created, and checked random samples by comparing both diffs with openclaw/openclaw main at e3415ea8c55 (model-assisted, blind to the score; a second independent blind pass on 10 pairs agreed on 9 of them and on every "already landed" verdict):
| Score of the top candidate | Open PRs | Sampled | Already landed by that PR | Partly landed |
|---|---|---|---|---|
| 0.85 or above | 53 | 25 | 11–12 of 25 (44–48%; Wilson 95% CI 27–67%) | 7–8 |
| 0.80 to 0.85 | 126 | 25 | 3 of 25 (12%; 4–30%) | 7 |
One pair in the upper band is borderline between "landed" and "partly landed", hence the range. In two more pairs of the upper band, a different pull request had already landed the change. These rates depend on the embedding model and basis above; with the default text-embedding-3-small they may differ.
How far the filters can be trusted
The archive keeps current values, not history. A merge time never changes, so --merged-after is exact up to state_as_of and can only miss merges after it. Reopening clears closed_at, so --open-at can only include extra rows (a row with an earlier close and reopen), never drop one that was open. These directions, and each row's state, assume the closed-thread sweeps have covered the archive since its rows were stored. On the copy above, 0 of 1,100 checked --open-at rows were wrong and all 1,100 checked merged_at values matched GitHub's mergedAt.
Activity
clawsweeper commentedon Oct 10, 2026
Codex review: this still needs some work.
Summary
This issue remains necessary: current main lacks the requested capability, and an open same-author PR explicitly owns its implementation.
Reproducibility: Current source establishes the missing filters and metadata and vector decoding before result limiting; the reported timings and SQL-variable threshold were not independently reproduced.
Root-cause cluster
Relationship:
canonicalCanonical: #238
Summary: This issue owns the temporal neighbor-selection request and has one explicit open implementation candidate.
Members:
fixed_by_candidate: feat(neighbors): filter by state and merge time before ranking #239 - The open same-author PR explicitly closes this issue and proposes its filters, metadata and freshness output.Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.
Maintainer decision needed
Next step
Review #239 and resolve the temporal-filter and freshness contract before resolving this issue.
Review details
Best possible solution:
Provide approved SQL candidate filters before vector decoding and ranking, with additive metadata and explicit historical-state limitations.
Do we have a high-confidence way to reproduce the issue?
Current source establishes the missing filters and metadata and vector decoding before result limiting; the reported timings and SQL-variable threshold were not independently reproduced.
Is this the best way to solve the issue?
Filtering the existing SQL candidate query directly addresses the workload without a storage migration, subject to agreement on temporal and freshness semantics.
AGENTS.md: found, but no applicable review policy affected this item.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against efe01cf9019a.
Label changes
Label changes:
No label changes.
Label justifications:
P2: Large-archive cleanup searches incur substantial extra work, while the existing workaround limits urgency.impact:ux-friction: Identifying later-merged candidates requires broad neighbor retrieval and a separate metadata join.Evidence reviewed
What I checked:
internal/cli/neighbors.go:18, efe01cf9019a)internal/store/vectors.go:63, efe01cf9019a)docs/clustering.md:118, efe01cf9019a)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed October 16, 2026, 7:59 AM ET / 11:59 UTC.