Compatibility issue between s3fs, fsspec, and datasets

[njbrake]

Describe the bug

Here is the full error stack when installing:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible.
Successfully installed aiobotocore-2.12.1 aioitertools-0.11.0 botocore-1.34.51 fsspec-2024.3.1 jmespath-1.0.1 s3fs-2024.3.1 urllib3-2.0.7 wrapt-1.16.0

When I install with pip, pip allows this error to exist while still installing s3fs, but this error breaks poetry, since poetry will refuse to install s3fs because of the dependency conflict.

Maybe I'm missing something so maybe it's not a bug but some mistake on my end? Any input would be helpful. Thanks!

Steps to reproduce the bug

1. conda create -n tmp python=3.10 -y
2. conda activate tmp
3. pip install datasets
4. pip install s3fs

Expected behavior

I would expect there to be no error.

Environment info

MacOS (ARM), Python3.10, conda 23.11.0.

[mariosasko]

Hi! Instead of running pip install separately for each package, you should pass all the packages to a single pip install call (in this case, pip install datasets s3fs) to let pip properly resolve their versions.

[njbrake]

Hi! Instead of running pip install separately for each package, you should pass all the packages to a single pip install call (in this case, pip install datasets s3fs) to let pip properly resolve their versions.

Thanks so much! My inexperience with pip is showing 😆 🙈

[poult-lab]

Hi! Instead of running pip install separately for each package, you should pass all the packages to a single pip install call (in this case, pip install datasets s3fs) to let pip properly resolve their versions.

you are awesome bro

[Konstantina-Lazaridou]

Hey, the suggestion by @mariosasko unfortunately only address this issue via pip. The original message was about poetry and I am still facing a dependency conflict with that.

The following command complains first about fsspec (... no versions of fsspec match ...) and then I get an error.

Command:
poetry add datasets s3fs

Error:
... your project ... depends on both datasets (^3.1.0) and s3fs (^2024.10.0), version solving failed

Installing first s3fs and then the rest of the huggingface libraries, like datasets, also did not help.