Skip to content

fix: add id lock to fix map leak - #241

Open
ningmingxiao wants to merge 1 commit into
containerd:mainfrom
ningmingxiao:fix_leak
Open

ningmingxiao wants to merge 1 commit into
containerd:mainfrom
ningmingxiao:fix_leak

Conversation

@ningmingxiao

@ningmingxiao ningmingxiao commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

fixes / addresses containerd/containerd#13561

@marcoabreu

Copy link
Copy Markdown

Independent reproduction, in case it helps this along.

We hit this on Docker hosts where containers carry short-interval HEALTHCHECKs — every healthcheck is a docker exec, so the workload lands squarely on the raced streams.Store / streams.Delete path.

What the data looked like:

  • Shim memory tracks cumulative exec count, not container uptime or log volume. A container with a 3s healthcheck interval reached ~96 MB RSS after ~2.05M execs over 71 days; containers with no healthcheck on the same hosts stayed at a ~6 MB baseline after 78 days.
  • It is real private heap, not shared-text RSS inflation: smaps_rollup showed PSS 96.2 MB against RSS 98.9 MB, with Anonymous at 95.5 MB. Under genuine memory pressure the kernel reclaimed only ~5 MB of it.
  • Not Go's MADV_FREE behaviour — the shims were built with go1.23.7 and go1.25.8.
  • Observed on containerd 1.7.27 and 2.2.x.
  • Per-exec cost varied roughly 10–50 bytes across hosts, which is consistent with a race whose window depends on CPU speed and load rather than a fixed per-call leak.

The only effective mitigations we found were lengthening the healthcheck interval or dropping the exec-based healthcheck entirely. Restarting dockerd or containerd does not release it, since shims outlive both.

Would be good to see this land.

@ningmingxiao

Copy link
Copy Markdown
Contributor Author

ping @fuweid

@ningmingxiao

Copy link
Copy Markdown
Contributor Author

ping @mikebrow @mxpv can you take a loook thanks

@mikebrow

Copy link
Copy Markdown
Member

needs rebase

@ningmingxiao

ningmingxiao commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor Author

done thanks @mikebrow

Comment thread server.go Outdated
}
return nil
}
lockID := fmt.Sprintf("%d", id)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use strconv.Itoa for these (here, and the other places it was added)? Saves some overhead, and will keep linters at bay.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done @thaJeztah

Signed-off-by: 宁明晓10296073 <ning.mingxiao@zte.com.cn>
Signed-off-by: ningmingxiao <ning.mingxiao@zte.com.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants