Skip to content

fix(build): add jitter and delay cap to cosign retry backoff - #598

Merged
bedatty merged 2 commits into
developfrom
fix/cosign-retry-jitter-and-cap
Jul 21, 2026
Merged

bedatty merged 2 commits into
developfrom
fix/cosign-retry-jitter-and-cap

Conversation

@bedatty

@bedatty bedatty commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Description

Adds jitter and a delay cap to the cosign signing retry policy, and raises the default max attempts, to better absorb Rekor outages that can last several minutes.

The 2.2.0-beta.6 release incident showed the current retry policy is insufficient: attempt #1 failed because cosign sign hit a transient Rekor 404 getLogEntryByUuidNotFound across all 3 retry attempts.

Changes:

  • src/security/cosign-sign/action.yml: max-attempts default raised from 3 to 5; new max-delay input (default 60s) caps the exponential backoff; retry delay is now randomized with equal jitter (half fixed + half random) to avoid multiple concurrent jobs retrying against Rekor at the same instant.
  • .github/workflows/build.yml: cosign_max_attempts default raised to 5; new cosign_max_delay input (default 60) wired through to the composite action.
  • src/security/cosign-sign/README.md and docs/build-workflow.md: updated to document the new defaults, the cap, and the jitter behavior.

Type of Change

  • fix: Bug fix in a workflow (incorrect behavior, broken step, wrong condition)

Breaking Changes

None. Defaults only get more resilient (more attempts, capped/jittered backoff); initial-delay/cosign_initial_delay are unchanged at 5s.

Testing

  • YAML syntax validated locally
  • Triggered a real workflow run on a caller repository using @this-branch or the beta tag
  • Verified all existing inputs still work with default values
  • Confirmed no secrets or tokens are printed in logs
  • Checked that unrelated workflows are not affected

Related Issues

Closes #450

Raise cosign_max_attempts default from 3 to 5, cap the exponential
backoff at cosign_max_delay (default 60s), and randomize it (equal
jitter) to avoid thundering-herd retries against Rekor.

Closes #450
@bedatty
bedatty requested a review from a team as a code owner July 21, 2026 13:52
@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 515fc8fd-bd20-48c4-9a78-5ebae39e671c

📥 Commits

Reviewing files that changed from the base of the PR and between ebdcfb6 and f10ae9a.

📒 Files selected for processing (3)
  • docs/build-workflow.md
  • src/security/cosign-sign/README.md
  • src/security/cosign-sign/action.yml

Summary by CodeRabbit

  • New Features
    • Added configurable signing retry delay capping via a new max-delay setting.
    • Signing retries now use capped exponential backoff with equal-jitter to reduce retry bursts.
    • Increased the default number of signing attempts from 3 to 5.
  • Documentation
    • Updated build workflow and signing action guidance to describe max-delay, retry timing (growth and cap), and updated default behavior.

Walkthrough

The cosign action now supports capped equal-jitter exponential retries, defaults to five attempts, and exposes a 60-second delay cap. The reusable build workflow forwards this setting, and related documentation describes the updated retry policy.

Changes

Cosign retry resilience

Layer / File(s) Summary
Action retry contract and implementation
src/security/cosign-sign/action.yml, src/security/cosign-sign/README.md
Adds max-delay, raises the default attempts from 3 to 5, wires the delay cap into the signing step, and applies capped equal-jitter retry sleeps.
Workflow retry configuration
.github/workflows/build.yml
Adds the cosign_max_delay workflow input, raises cosign_max_attempts to 5, and forwards the delay cap to the cosign action.
Retry policy documentation
docs/build-workflow.md
Documents the new delay cap, five-attempt default, exponential backoff, and jitter behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change to cosign retry backoff.
Description check ✅ Passed The description covers the change, affected files, type, breaking changes, testing, and related issue.
Linked Issues check ✅ Passed The PR adds jitter, raises max-attempts, adds a delay cap, and updates build defaults as requested by #450.
Out of Scope Changes check ✅ Passed No unrelated code changes are evident beyond the retry policy, docs, and workflow wiring required by the issue.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/cosign-retry-jitter-and-cap

Comment @coderabbitai help to get the list of available commands.

@lerian-studio lerian-studio added size/XS PR changes < 50 lines documentation Improvements or additions to documentation workflow Changes to one or more reusable workflow files security Changes to security workflows or vulnerability reporting policy composite Changes to any composite action manifest (src/**/*.yml) labels Jul 21, 2026
@lerian-studio

lerian-studio commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

🛡️ CodeQL Analysis Results

Languages analyzed: actions

Found 1 issue(s): 1 Medium

Severity Rule File Message
🟡 Medium actions/untrusted-checkout/medium .github/workflows/build.yml:300 Potential unsafe checkout of untrusted pull request on non-privileged workflow.

🔍 View full scan logs | 🛡️ Security tab

@lerian-studio

lerian-studio commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

🔍 PR Validation Summary

✅ PR Mergeable — no blocking failures

Check Status Blocking
Source Branch ✅ success yes
PR Title ✅ success yes
PR Description ✅ success yes
PR Size ✅ success no
Auto Labels ✅ success no
PR Metadata ✅ success no

🔍 View workflow run

@lerian-studio

lerian-studio commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Lint Analysis

Check Files Scanned Status
YAML Lint 2 file(s) ✅ success
Action Lint 1 file(s) ✅ success
Pinned Actions 2 file(s) ✅ success
Markdown Link Check 2 file(s) ✅ success
Spelling Check 4 file(s) ✅ success
Shell Check 2 file(s) ✅ success
README Check 2 file(s) ✅ success
Composite Schema 1 file(s) ✅ success
Deployment Matrix no changes ⏭️ skipped

🔍 View full scan logs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.

Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.

👉 Steps to fix this

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/security/cosign-sign/action.yml`:
- Around line 24-27: Validate max-attempts, initial-delay, and max-delay as
non-negative integers before entering the retry loop, rejecting invalid values
before they reach seq, -gt, or arithmetic expressions. Update the input-handling
logic associated with the action’s retry configuration while preserving the
existing defaults and backoff behavior for valid values.
- Around line 17-19: Revert the retry defaults from 5 to 3 in both caller-facing
locations: src/security/cosign-sign/action.yml lines 17-19 and
.github/workflows/build.yml lines 159-161. Update the default associated with
the cosign signing retry configuration at each site; no other retry behavior
changes are needed.
- Around line 115-123: Update the retry backoff logic near capped_delay so the
persistent delay value cannot exceed MAX_DELAY before the next iteration. After
calculating the next delay, clamp delay to MAX_DELAY or skip multiplication when
it would exceed the ceiling, while preserving the existing capped sleep and
jitter behavior.

In `@src/security/cosign-sign/README.md`:
- Line 89: The configuration examples must demonstrate the new delay-cap
overrides: update src/security/cosign-sign/README.md lines 89-89 to include
max-delay in the action example, and docs/build-workflow.md lines 285-285 to
include cosign_max_delay in the workflow example.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 474d8e8c-928c-4ab3-bc65-223914d2a698

📥 Commits

Reviewing files that changed from the base of the PR and between 67dad77 and ebdcfb6.

📒 Files selected for processing (4)
  • .github/workflows/build.yml
  • docs/build-workflow.md
  • src/security/cosign-sign/README.md
  • src/security/cosign-sign/action.yml

Comment thread src/security/cosign-sign/action.yml
Comment thread src/security/cosign-sign/action.yml
Comment thread src/security/cosign-sign/action.yml
Comment thread src/security/cosign-sign/README.md
Clamp the persistent delay variable to max-delay (not just the sleep
value) to avoid unbounded growth across retries, and add max-delay to
the README/docs configuration examples.
@lerian-studio lerian-studio added size/S PR changes 50–199 lines and removed size/XS PR changes < 50 lines labels Jul 21, 2026
@bedatty
bedatty merged commit c611ab9 into develop Jul 21, 2026
2 checks passed
@github-actions
github-actions Bot deleted the fix/cosign-retry-jitter-and-cap branch July 21, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

composite Changes to any composite action manifest (src/**/*.yml) documentation Improvements or additions to documentation security Changes to security workflows or vulnerability reporting policy size/S PR changes 50–199 lines workflow Changes to one or more reusable workflow files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build: cosign retry resilience — add jitter and raise max-attempts default

2 participants