Skip to content

CertBot cron jobs are failing - Leaving certificates to expire. #325

Closed
Closed
CertBot cron jobs are failing - Leaving certificates to expire.#325
Assignees
WadeBarnes

Description

@WadeBarnes
WadeBarnes
opened on Apr 7, 2026

We've had a significant number of CertBot cron job failures recently. These failures are leaving site certificates to expire.

The issue appears to be one of timing. It seems that in some cases the containers are failing to start in a timely fashion (throttling messages can be seen in the logs) while other times the route(s) generated by the container are not ready in time for the authentication callbacks.

Manually triggering the job(s) after updating the configuration with settings that slow down the process of creating the routes and triggering the authentication callbacks allow the process to complete successfully.

Latest failures:

  • a99fd4-dev
  • a99fd4-test
  • ca7f8f-dev
  • ca7f8f-prod
  • ca7f8f-test
  • e79518-dev
  • e79518-prod
  • e79518-test

Activity

  1. WadeBarnes
    self-assigned this
    on Apr 7, 2026
  2. github-project-automation
    added this to CDT Enterprise Appson Apr 7, 2026

  3. WadeBarnes commented on Apr 7, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 7, 2026
    MemberAuthor

    startingDeadlineSeconds is set to 60 (1 minute) seconds
    activeDeadlineSeconds is set to 300 (5 minutes) seconds

    The startup active deadline appears to be the issue: Job was active longer than specified deadline

    So not the issue first thought.

  4. WadeBarnes commented on Apr 7, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 7, 2026
    MemberAuthor

    Throttling issues and therefore exceeding the active deadline could be due to the low CPU limit set on the cron jobs:

    resources:
  limits:
    cpu: 250m
  requests:
    cpu: 50m

  5. WadeBarnes commented on Apr 7, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 7, 2026
    MemberAuthor

    Updating resources to:

    resources:
  requests:
    cpu: 100m
  6. WadeBarnes
    moved this to In Review in CDT Enterprise Appson Apr 7, 2026

  7. WadeBarnes commented on Apr 7, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 7, 2026
    MemberAuthor

    Now that all jobs are updated, we'll monitor for a few runs to see if that resolves the issues.

  8. WadeBarnes commented on Apr 13, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 13, 2026
    MemberAuthor

    Jobs failed in:

    • a99fd4-dev
    • ca7f8f-dev
    • ca7f8f-test
    • ca7f8f-prod
    • e79518-prod

    Increased activeDeadlineSeconds to 600 seconds (10 minutes). Updated the remaining environments to match.

  9. WadeBarnes commented on Apr 16, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 16, 2026
    MemberAuthor

    Most of the jobs failed again yesterday. Updated startingDeadlineSeconds to 300 seconds (5 minutes), to see if that helps.

  10. WadeBarnes commented on Apr 21, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 21, 2026
    MemberAuthor

    All jobs for a99fd4 completed successfully on the 19th.
    All jobs for ca7f8f completed successfully on the 19th.
    All jobs for e79518 completed successfully on the 19th.

    However the run in e79518-test failed to renew a certificate on the 19th, however a follow-p manual run the following day renewed the certificate without issue.

    It appears the route was not ready - query resulted in a 404.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
networkpolicy.networking.k8s.io/certbot-allow-ingress created
service/certbot created
route.route.openshift.io/acme-challenge-tails-test.vonx.io created
...
Renewing an existing certificate for tails-test.vonx.io
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: tails-test.vonx.io
Type: unauthorized
Detail: 142.34.194.118: Invalid response from https://tails-test.vonx.io/.well-known/acme-challenge/<redacted>: 404
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 8080. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.
Failed to renew certificate tails-test.vonx.io with error: Some challenges have failed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  11. WadeBarnes commented on Apr 23, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 23, 2026
    MemberAuthor

    Failed jobs - Exceeded active deadline (which is 10 minutes now):

    • e79518-dev
    • e79518-test
    • e79518-prod
    • a99fd4-dev
    • a99fd4-test
    • ca7f8f-test

    Other environments succeeded, and there were now certificates to update.

  12. WadeBarnes commented on Apr 23, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 23, 2026
    MemberAuthor

    While running the failed jobs manually I was able to capture what a script hang looks like. The key piece being error: no objects passed to create.

    I0423 12:30:18.575724      14 request.go:665] Waited for 1.01540256s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/monitoring.rhobs/v1?timeout=32s
CERTBOT_DEBUG = false
Deleting ACME challenge resources ...
I0423 12:30:21.679155      56 request.go:665] Waited for 1.001373351s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/performance.openshift.io/v1alpha1?timeout=32s
I0423 12:30:31.697117      56 request.go:665] Waited for 1.218326596s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/quota.openshift.io/v1?timeout=32s
No resources found
networkpolicy.networking.k8s.io/certbot-allow-ingress created
service/certbot created
error: no objects passed to create
Manage individual certificates for each unique host.
No new certificate(s) have been issued
Deleting ACME challenge resources ...
I0423 12:30:43.442442     184 request.go:665] Waited for 1.0195646s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/postgres-operator.crunchydata.com/v1beta1?timeout=32s
I0423 12:30:53.460872     184 request.go:665] Waited for 1.237014989s due to client-side throttling, not priority and fairness, request: GET:https://10.98.0.1:443/apis/pipelinesascode.tekton.dev/v1alpha1?timeout=32s
service "certbot" deleted
networkpolicy.networking.k8s.io "certbot-allow-ingress" deleted

    https://console.apps.silver.devops.gov.bc.ca/k8s/ns/a99fd4-dev/cronjobs/certbot/jobs

  13. WadeBarnes commented on Apr 23, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 23, 2026
    MemberAuthor

    I'll integrate a fix for such issues into the work here; bcgov/certbot#38

  14. WadeBarnes commented on Apr 23, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 23, 2026
    MemberAuthor

    Same thing is happening with the job in https://console.apps.silver.devops.gov.bc.ca/k8s/ns/a99fd4-test/cronjobs/certbot/

    Seems to be an issue mostly when there is nothing to do. i.e. There are no routes that are being maintained by certbot. The script hangs on the cleanup phase. Which makes sense as to why it runs when CERTBOT_DELETE_ACME_ROUTES is set to false.

  15. WadeBarnes
    moved this from In Review to In Progress in CDT Enterprise Appson Apr 23, 2026

  16. 1 remaining item

  17. WadeBarnes commented on Apr 28, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 28, 2026
    MemberAuthor

    Failed jobs:

    • 4a9599-prod

    Running the job manually, the pod is having issues mounting it's PVC.

    Image

    Deleted the job and restarted. Second attempt succeeded without error.

    Wonder if this is part of the problem - pods have issues mounting their PVCs and exceed their deadlines.

  18. WadeBarnes commented on Apr 30, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 30, 2026
    MemberAuthor

    Failed to start/complete:

    • e79518-dev
    • e79518-test
    • e79518-prod
    • ca7f8f-dev
    • ca7f8f-test
    • ca7f8f-prod
    • a99fd4-test
    • 4a9599-prod

  19. WadeBarnes commented on Apr 30, 2026

    @WadeBarnes
    WadeBarnes
    on Apr 30, 2026
    MemberAuthor

    Changed restartPolicy: Never to restartPolicy: OnFailure

  20. WadeBarnes commented on May 1, 2026

    @WadeBarnes
    WadeBarnes
    on May 1, 2026
    MemberAuthor

    Failed to start/complete:

    • e79518-prod
    • ca7f8f-prod
    • 4a9599-prod

    The jobs don't seem to be retrying even though the restart policy is set.

  21. WadeBarnes commented on May 4, 2026

    @WadeBarnes
    WadeBarnes
    on May 4, 2026
    MemberAuthor

    Failed to start/complete (May 1st):

    • ca7f8f-test
    • a99fd4-test

  22. WadeBarnes commented on May 4, 2026

    @WadeBarnes
    WadeBarnes
    on May 4, 2026
    MemberAuthor

    Failed to start/complete:

    • ca7f8f-dev
    • ca7f8f-test
    • ca7f8f-prod
    • a99fd4-dev
    • a99fd4-test
    • e79518-dev
    • e79518-test
    • 4a9599-prod

    Ran and passed, clearing previous failure:

    • ca7f8f-test

  23. WadeBarnes commented on May 4, 2026

    @WadeBarnes
    WadeBarnes
    on May 4, 2026
    MemberAuthor

    Steven recommending we try a different schedule. They are seeing issues when a bunch of cron jobs all fire at once.

    Current schedule is 0 0 * * 1,4 - Which fires Sunday and Wednesday at 5pm - Pacific.

  24. WadeBarnes commented on May 4, 2026

    @WadeBarnes
    WadeBarnes
    on May 4, 2026
    MemberAuthor

    Updated schedule to 30 8 * * 1,4 - “At 08:30 on Monday and Thursday.” (UTC), “At 01:30 on Monday and Thursday.” (Pacific)

  25. WadeBarnes commented on May 5, 2026

    @WadeBarnes
    WadeBarnes
    on May 5, 2026
    MemberAuthor

    Failed to start/complete:

    • ca7f8f-prod
    • 4a9599-prod

    These were run on the old schedule, not the new schedule. Seems the failure notifications were delayed - new in my inbox, but not recent runs.

  26. WadeBarnes commented on May 6, 2026

    @WadeBarnes
    WadeBarnes
    on May 6, 2026
    MemberAuthor

    Important detail of the email notifications. If you don't clear failures you will get notified of ANY existing failures every morning. Failure notifications are not just for new failures.

  27. WadeBarnes commented on May 8, 2026

    @WadeBarnes
    WadeBarnes
    on May 8, 2026
    MemberAuthor

    All cron jobs succeeded on the new schedule. Will monitor a couple more runs to see if this is a stable state.

  28. WadeBarnes commented on May 11, 2026

    @WadeBarnes
    WadeBarnes
    on May 11, 2026
    MemberAuthor

    All cron jobs succeeded on the new schedule this morning.

  29. WadeBarnes commented on May 14, 2026

    @WadeBarnes
    WadeBarnes
    on May 14, 2026
    MemberAuthor

    All cron jobs succeeded on the new schedule this morning. Closing this ticket and this issue has been resolved.

    Summary:

    • Cron jobs were failing to mount their PVCs due to the current concurrency and scalability issues with the platform infrastructure (fixes are supposed to be coming in June).
    • The failure logs were misleading - they made it look if the Cron Job was starting, but taking too long to complete, which was not the case at all.
    • Moving the Cron jobs to a different schedule fixed the issue - by avoiding the actual issue. This is a workaround not an actual fix.
  30. WadeBarnes
    closed this as completedon May 14, 2026
  31. github-project-automation
    moved this from In Progress to Complete in CDT Enterprise Appson May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions