Dask installation failing on newest package

[drewaogle]

When working on a change for athena, pip started failing.
I traced the error to pip selecting an old dask ( 0.8, I think ) which failed to install because it used a word that became reserved in python 3.7.

Note - this fails only with updated pip (24.2)

pip3 install     dask ffmpeg-python     opencv-python     pandas==1.5.3     numpy     dbeval     requests     boto3     jsonschema    aperturedb==0.4.33 pre-commit

is the line that fails.

pip3 install     dask ffmpeg-python     opencv-python     pandas==1.5.3     numpy     dbeval     requests     boto3     jsonschema    aperturedb==0.4.32 pre-commit

works

but also -

pip3 install     dask>=2022.10.2 ffmpeg-python     opencv-python     pandas==1.5.3     numpy     dbeval     requests     boto3     jsonschema    aperturedb==0.4.33 pre-commit

works.

The fact that I was able to nudge dask to choose a newer version and it then selected the newest makes me think it is a problem with the pip resolver, which has had known issues in the past. Additionally, when you run the top, it shows that pip ends up panicking and searching through a lot of dasks for some reason.

[drewaogle]

Also if you run it in a vm, you can notice the pip process maxing cpu - which shouldn't be happening, and this is when it is trying to determine the "best" version.

[gsaluja9]

Where is this combination restricting.
Especially the exact pandas version == 1.5.3

This does not repro in our CI, so lets add some more light to it.