Skip to content

TestGroupKWFPoolDestroyedDropsFromPlacer flakes: placer picks a PoolMissing pool #198

Open

Description

@kvaps

TestGroupKWFPoolDestroyedDropsFromPlacer fails intermittently: the placer puts a diskful replica on the node whose pool was just marked PoolMissing.

--- FAIL: TestGroupKWFPoolDestroyedDropsFromPlacer (8.87s)
    group_k_test.go:473: placer landed diskful replica on node worker-1 with PoolMissing pool lvm-thin ...

It predates #193. Local runs, 56 iterations on each side:

That is not enough to tell whether the rate changed, but it does show the failure is older than #193.

The test waits for Status.PoolMissing through the direct client (group_k_test.go:435) and then autoplaces right away. My guess is the placer reads storage pools through the manager's cache, which can still hold the pool without PoolMissing. If so, either the test races the informer, or the placer has a real window after a pool goes missing where it still picks it. The second is the Bug 83 case on a real cluster too, so it's worth checking which one it is before making the test wait longer.

To reproduce:

go test -tags=integration ./tests/integration/ -run 'TestGroupKWFPoolDestroyedDropsFromPlacer$' -count=20

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions