Skip to content

Deduplicate registered informers by their underlying informers - #3957

Draft
grokspawn wants to merge 1 commit into
operator-framework:masterfrom
grokspawn:fix-runinformers-informer-race
Draft

grokspawn wants to merge 1 commit into
operator-framework:masterfrom
grokspawn:fix-runinformers-informer-race

Conversation

@grokspawn

Copy link
Copy Markdown
Contributor

Summary

registerInformer deduplicates informers by interface identity. That is no longer sufficient with client-go's typed informer factories: every Informer() call returns a new typedSharedIndexInformer wrapper around the same factory-cached underlying informer, and two such wrappers of one informer compare unequal. When a caller registers two wrappers of one informer, both are appended and RunInformers runs the underlying informer twice concurrently.

sharedIndexInformer.RunWithContext does not tolerate a double Run: the HasStarted() guard is checked before the construction closure that sets s.started, so two concurrent Runs both proceed, racing s.controller and FIFO construction against the stop path, and both Runs' shutdown goroutines close(s.synced) — ending in:

panic: close of closed channel

Observed downstream (OpenShift CI, go test -race -count=1): a downstream plugin registers a second wrapper of the namespace and CSV informer groups, and the pkg/controller/operators/olm suite intermittently fails with three data races between RunWithContext frames followed by that panic:

https://prow.ci.openshift.org/view/gs/test-platform-results-public/pr-logs/pull/openshift_operator-framework-olm/1396/pull-ci-openshift-operator-framework-olm-main-unit-olm/2115505456544747520

Stack signature of the double Run (two Run goroutines created by the go informer.Run(...) loop in RunInformers, both inside typedSharedIndexInformer[*corev1.Namespace].Run):

Write:  NewRealFIFOWithOptions / cache.New  <- RunWithContext.func1 (shared_informer.go:854/873)
Read:   RealFIFO.Done / HasSyncedChecker    <- RunWithContext.func2 (shared_informer.go:893)
Panic:  close of closed channel             <- RunWithContext.func2 (shared_informer.go:895)

Fix

Deduplicate on the underlying informer rather than wrapper identity: every sharedIndexInformer owns its indexer from construction, so informers backed by the same underlying informer report the same GetIndexer() value. Registration now skips an informer whose indexer is already registered.

Verified against the failing downstream configuration: with this change, an operator whose setup registers two wrappers each of the namespace and CSV informers registers 17 informers with 17 unique underlying informers (previously 19 registered / 17 unique), and the duplicate Run — and with it the race and panic — cannot occur.

Test

TestRegisterInformerDeduplicatesTypedWrappers builds one cache.NewSharedIndexInformer, wraps it twice with cache.NewTypedSharedIndexInformer, registers both wrappers and asserts a single registration.

Notes

  • The reproducing double registration currently exists only in downstream carry code, but the weakness is general: obtaining Informer() twice from one typed informer group and registering each result is a natural use of the client-go API surface.
  • sharedIndexInformer.RunWithContext's check-then-set start guard could also be hardened in client-go, but running an informer twice is caller error; deduplicating at registration is the correct fix on our side.

registerInformer deduplicates by interface identity, but client-go's
typed informer factories return a new typedSharedIndexInformer wrapper
for every Informer() call around the same cached underlying informer,
and two such wrappers of one informer compare unequal. When callers
obtain Informer() more than once from the same typed informer group and
register each result, both wrappers are appended and RunInformers runs
the underlying informer twice concurrently.

sharedIndexInformer.RunWithContext does not tolerate that: its
HasStarted guard is checked before the construction closure that sets
s.started, so two concurrent Runs both proceed, racing the controller
and FIFO construction against the stop path and eventually panicking
with "close of closed channel" when both Runs close s.synced.

Deduplicate on the underlying informer instead: every
sharedIndexInformer owns its indexer from construction, so informers
backed by the same underlying informer report the same indexer.

Observed downstream, where a plugin registers a second wrapper of the
namespace and CSV informers and the unit suite (go test -race)
intermittently fails with three data races between RunWithContext
frames followed by that panic:
https://prow.ci.openshift.org/view/gs/test-platform-results-public/pr-logs/pull/openshift_operator-framework-olm/1396/pull-ci-openshift-operator-framework-olm-main-unit-olm/2115505456544747520

Signed-off-by: grokspawn <jordan@nimblewidget.com>
@openshift-ci

openshift-ci Bot commented Nov 2, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 2, 2026
@openshift-ci

openshift-ci Bot commented Nov 2, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign perdasilva for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Nov 2, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant