Conversation
…ports
processUpdates logs an error if a volume a node reported as unpublished
can't be found in the store, but then carries on and dereferences it:
volume := store.GetVolume(tx, volumeID)
if volume == nil {
logger.Error("volume unavailable")
}
...
for _, status := range volume.PublishStatus {
UpdateVolumeStatus doesn't check that the volume exists; it only queues
the report until the next batch. A volume may be removed from the store
before the report is processed, for example when it is force-removed
while a node is still unpublishing it, as RemoveVolume with Force set
deletes the volume regardless of its PublishStatus. When that happens,
the dispatcher panics with a nil pointer dereference, taking down the
leader manager.
Return early if the volume doesn't exist, as is already done for nodes
and tasks in the same function. There is nothing to update for a volume
that no longer exists, and returning nil from the callback does not
affect updates to other volumes in the batch.
Closes: moby#3208
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3301 +/- ##
==========================================
+ Coverage 14.71% 14.75% +0.04%
==========================================
Files 198 198
Lines 93035 93034 -1
==========================================
+ Hits 13690 13731 +41
+ Misses 78011 77958 -53
- Partials 1334 1345 +11 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes #3208
- What I did
Fixed a nil pointer dereference in
(*Dispatcher).processUpdates. When a node reports a volume as unpublished, and that volume is gone from the store by the time the report is processed, the leader manager panics.UpdateVolumeStatusonly queues the report until the next batch and doesn't check that the volume exists.processUpdatesthen logsvolume unavailablebut carries on and ranges overvolume.PublishStatus.One way to hit this: the volume is force-removed while a node is still unpublishing it.
RemoveVolumewithForcedeletes the volume regardless of itsPublishStatus. For example, a user removes a service,docker volume rmfails with "volume is still in use", and they retry with--force. The node then finishes unpublishing and reports it.- How I did it
Return early from the batch callback if the volume doesn't exist. The same function already does this for nodes and tasks, and so do the scheduler (
freeVolumes) and the CSI manager for missing volumes. There is nothing to update for a volume that no longer exists. Returningnilonly skips that volume, so the rest of the batch is still processed.- How to test it
The new test puts two volumes in
PENDING_NODE_UNPUBLISHand queues unpublish reports for both. It then deletes one volume from the store and callsprocessUpdates. It checks that there is no panic and that the other volume still moves toPENDING_UNPUBLISH. Without the fix it fails withinvalid memory address or nil pointer dereference.The test doesn't start the dispatcher's
Runloop on purpose. The loop processes updates every 100ms and could handle the reports before the volume is deleted, and then the test would pass without the fix.- Description for the changelog
Fix a manager panic when a volume is removed before a node's unpublish report for it is processed.