Skip to content

manager/dispatcher: skip missing volumes when processing unpublish reports - #3301

Open
grey3228 wants to merge 1 commit into
moby:masterfrom
grey3228:dispatcher-fix-missing-volume
Open

grey3228 wants to merge 1 commit into
moby:masterfrom
grey3228:dispatcher-fix-missing-volume

Conversation

@grey3228

Copy link
Copy Markdown

fixes #3208

- What I did

Fixed a nil pointer dereference in (*Dispatcher).processUpdates. When a node reports a volume as unpublished, and that volume is gone from the store by the time the report is processed, the leader manager panics.

UpdateVolumeStatus only queues the report until the next batch and doesn't check that the volume exists. processUpdates then logs volume unavailable but carries on and ranges over volume.PublishStatus.

One way to hit this: the volume is force-removed while a node is still unpublishing it. RemoveVolume with Force deletes the volume regardless of its PublishStatus. For example, a user removes a service, docker volume rm fails with "volume is still in use", and they retry with --force. The node then finishes unpublishing and reports it.

- How I did it

Return early from the batch callback if the volume doesn't exist. The same function already does this for nodes and tasks, and so do the scheduler (freeVolumes) and the CSI manager for missing volumes. There is nothing to update for a volume that no longer exists. Returning nil only skips that volume, so the rest of the batch is still processed.

- How to test it

go test ./manager/dispatcher/ -run TestVolumeUnpublishedDeletedVolume

The new test puts two volumes in PENDING_NODE_UNPUBLISH and queues unpublish reports for both. It then deletes one volume from the store and calls processUpdates. It checks that there is no panic and that the other volume still moves to PENDING_UNPUBLISH. Without the fix it fails with invalid memory address or nil pointer dereference.

The test doesn't start the dispatcher's Run loop on purpose. The loop processes updates every 100ms and could handle the reports before the volume is deleted, and then the test would pass without the fix.

- Description for the changelog

Fix a manager panic when a volume is removed before a node's unpublish report for it is processed.

…ports

processUpdates logs an error if a volume a node reported as unpublished
can't be found in the store, but then carries on and dereferences it:

    volume := store.GetVolume(tx, volumeID)
    if volume == nil {
        logger.Error("volume unavailable")
    }
    ...
    for _, status := range volume.PublishStatus {

UpdateVolumeStatus doesn't check that the volume exists; it only queues
the report until the next batch. A volume may be removed from the store
before the report is processed, for example when it is force-removed
while a node is still unpublishing it, as RemoveVolume with Force set
deletes the volume regardless of its PublishStatus. When that happens,
the dispatcher panics with a nil pointer dereference, taking down the
leader manager.

Return early if the volume doesn't exist, as is already done for nodes
and tasks in the same function. There is nothing to update for a volume
that no longer exists, and returning nil from the callback does not
affect updates to other volumes in the batch.

Closes: moby#3208
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 14.75%. Comparing base (22c2c3d) to head (893fc4f).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3301      +/-   ##
==========================================
+ Coverage   14.71%   14.75%   +0.04%     
==========================================
  Files         198      198              
  Lines       93035    93034       -1     
==========================================
+ Hits        13690    13731      +41     
+ Misses      78011    77958      -53     
- Partials     1334     1345      +11     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

potential nil ptr dereference in (*Dispatcher).processUpdates (SAST warning)

2 participants