feat: paginate delete blockers

[grantfitzsimmons]

Fixes #7515

Authored by @melton-jason

Description

This PR addresses a major performance bottleneck in Specify's delete blockers endpoint, which affects large databases. Currently, when fetching delete blockers for a resource, Specify attempts to load all referencing records at once into a single Python object using Django's Collector. For highly-referenced resources (particularly Agent, which is referenced by createdByAgent and modifiedByAgent across 515+ relationships) this eager collection can:

• Cause request timeouts as the server struggles to load tens or hundreds of thousands of related records
• Tie up Gunicorn workers, preventing them from responding to other incoming requests
• Exhaust system memory, triggering the Linux OOM killer and crashing the Specify process (reproducible locally with 8 GiB of RAM allocated to Docker)
• Cause memory thrashing as the kernel aggressively swaps, degrading performance across the entire instance

At least two production instances have experienced kernel-invoked process restarts due to this issue. Even when requests don't crash, they can hold the Python GIL long enough to block other threads.

Solution: Paginated delete blockers

This PR introduces pagination support for the delete blockers endpoint. Instead of loading all referencing records at once, related record IDs are fetched in configurable chunks (limit/offset). The new approach:

• Returns the first page of delete blockers in under 1 second for highly-referenced agents (previously timed out or crashed)
• Fetches all delete blockers in ~3 seconds when needed, still a dramatic improvement over Django's Collector
• Splits results into immediate blockers (PROTECT / protect_with_blockers) and deferred blockers (CASCADE), enabling the frontend to prioritize what's shown
• Preserves the original endpoint at a legacy URL (old_delete_blockers) for backward compatibility

Changes

New endpoint behavior (/api/specify/delete_blockers/<model>/<id>/):

• Accepts limit (default: 20) and offset (default: 0) query parameters
• Returns JSON with two arrays:
  • results: immediate blockers — relationships using PROTECT or protect_with_blockers
  • next: deferred blockers — relationships using CASCADE
• Each entry includes table, field, ids, offset, limit, and a complete flag indicating whether all matching records for that relationship were fetched
• Passing limit=0 fetches all results (unpaginated)

Future work

Further frontend changes are needed to adopt the paginated approach in the UI. Additional backend optimizations (e.g., query-level improvements) will be detailed in follow-up issues.

Developer Testing instructions

1. Paginated endpoint: GET /api/specify/delete_blockers/agent/<id>/?limit=20&offset=0 — should return the first 20 IDs per blocking relationship in under 1 second
2. complete flag: Verify complete: true when all related records fit within limit, and complete: false otherwise
3. Legacy compatibility: GET /api/specify/old_delete_blockers/<model>/<id>/ should return the full, unpaginated result set identical to the previous behavior
4. No blockers: A resource with zero references should return empty results and next arrays
5. Stress-test with a highly-referenced Agent in a database of comparable size to Geneva, Edinburgh, or Berlin to confirm no timeouts or OOM conditions occur

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch issue-7515-2

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[grantfitzsimmons]

@melton-jason Could you base this instead on v7_12_0_7_base?

[github-actions]

Warning

One or more dependencies are approaching or past End-of-Life.
Please plan upgrades accordingly.

STATUS=WARNING
NODE_VERSION=20
NODE_CYCLE=20
EOL_DATE=2026-04-30
DAYS_REMAINING=-156

--- Node.js ---
Version: 20
EOL: 2026-04-30
Status: WARNING

STATUS=OK
PYTHON_VERSION=3.12
PYTHON_CYCLE=3.12
EOL_DATE=2028-10-31
DAYS_REMAINING=759

--- Python ---
Version: 3.12
EOL: 2028-10-31
Status: OK

STATUS=WARNING
DJANGO_VERSION=4.2
DJANGO_CYCLE=4.2
EOL_DATE=2026-04-07
DAYS_REMAINING=-179

--- Django ---
Version: 4.2
EOL: 2026-04-07
Status: WARNING