Skip to content

[PULP-2474] Optimize Maven repository metrics counts - #541

Merged
dkliban merged 1 commit into
pulp:mainfrom
kdelee:pulp-2474-metrics-precompute
Oct 14, 2026
Merged

dkliban merged 1 commit into
pulp:mainfrom
kdelee:pulp-2474-metrics-precompute

Conversation

@kdelee

@kdelee kdelee commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

The repository metrics endpoint runs three separate COUNT(DISTINCT) queries over MavenPackage content. Combine them into one Django aggregate query using PostgreSQL row-valued DISTINCT expressions.

This preserves the API and exact results while reducing production-scale GABI latency from about 9.4–9.8s to 3.6–3.8s under three concurrent requests.

Tests:

  • Ruff format/check
  • 234 unit tests passed, 7 skipped
  • 2 focused functional tests passed in the OCI container

@kdelee
kdelee force-pushed the pulp-2474-metrics-precompute branch 2 times, most recently from e5062bb to eaec692 Compare October 14, 2026 18:07
Addresses PULP-2474.\nImplements the query consolidation suggested in pulp#539.
@kdelee
kdelee force-pushed the pulp-2474-metrics-precompute branch from eaec692 to 3ab63fa Compare October 14, 2026 18:07
@kdelee

kdelee commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

GABI benchmark compared the existing three-query implementation with the new single-query Django aggregate.

Existing SQL pattern:

  SELECT COUNT(*)
  FROM (
    SELECT DISTINCT group_id, artifact_id
    FROM maven_mavenpackage
    WHERE content_ptr_id IN (
      SELECT unnest(content_ids)
      FROM core_repositoryversion
      WHERE pulp_id = '<version>'
    )
  ) packages;

  SELECT COUNT(*)
  FROM (
    SELECT DISTINCT
      group_id,
      artifact_id,
      REGEXP_REPLACE(version, '-[^-]+$', '')
    FROM maven_mavenpackage
    WHERE content_ptr_id IN (
      SELECT unnest(content_ids)
      FROM core_repositoryversion
      WHERE pulp_id = '<version>'
    )
  ) versions;

  SELECT COUNT(*)
  FROM (
    SELECT DISTINCT group_id, artifact_id, version
    FROM maven_mavenpackage
    WHERE content_ptr_id IN (
      SELECT unnest(content_ids)
      FROM core_repositoryversion
      WHERE pulp_id = '<version>'
    )
  ) builds;

The API executed these as three separate database queries.

New SQL shape:

  SELECT
    COUNT(DISTINCT (group_id, artifact_id)),
    COUNT(DISTINCT (
      group_id,
      artifact_id,
      REGEXP_REPLACE(version, '-[^-]+$', '')
    )),
    COUNT(DISTINCT (group_id, artifact_id, version))
  FROM maven_mavenpackage
  WHERE content_ptr_id IN (
    SELECT unnest(content_ids)
    FROM core_repositoryversion
    WHERE pulp_id = '<version>'
  );

Results matched exactly.

Single-request results:

  Existing implementation:
    cold: ~4.86s for first query
    warm: ~3.0s per query
    ~9s total across all three queries

  New implementation:
    cold: 4.86s
    warm: 3.41s, 3.39s

Three concurrent requests:

  Existing implementation: 9.4–9.8s per request
  New implementation:       3.6–3.8s per request

Summary:

  Same counts
  One DB query instead of three
  Approximately 60% faster under concurrent polling
  Less database contention
  No API response or schema change

@kdelee kdelee changed the title Optimize Maven repository metrics counts [PULP-2474] Optimize Maven repository metrics counts Oct 14, 2026
@dkliban
dkliban merged commit cdfe60d into pulp:main Oct 14, 2026
24 of 27 checks passed
@patchback

patchback Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Backport to 0.34: 💚 backport PR created

✅ Backport PR branch: patchback/backports/0.34/cdfe60dee5cc2a112646d495b7660796dbc7fc6d/pr-541

Backported as #542

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants