ERROR: Could not find a version that satisfies the requirement while dockerizing a python project

Question

I am trying to dockerize a python project. The problem is that docker is unable to find many of the packages that I am importing from requirements.txt file.

My OS is Xubuntu 20.04. My docker version informations are as follow:

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:52:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

This is my dockerfile:

FROM pypy:3

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "pypy3", "./index.py" ]

The requirements.txt file is :

apturl==0.5.2
attrs==20.3.0
Automat==20.2.0
beautifulsoup4==4.9.3
blinker==1.4
blis==0.7.4
bs4==0.0.1
catalogue==2.0.3
catfish==1.4.13
certifi==2019.11.28
cffi==1.14.5
chardet==3.0.4
click==7.1.2
colorama==0.4.3
command-not-found==0.3
constantly==15.1.0
cryptography==3.4.7
cssselect==1.1.0
cupshelpers==1.0
cymem==2.0.5
dbus-python==1.2.16
defer==1.0.6
distro==1.4.0
distro-info===0.23ubuntu1
elasticsearch==7.13.0
entrypoints==0.3
h2==3.2.0
hpack==3.0.0
httplib2==0.14.0
hyperframe==5.2.0
hyperlink==21.0.0
idna==2.8
incremental==21.3.0
itemadapter==0.2.0
itemloaders==1.0.4
Jinja2==2.11.3
jmespath==0.10.0
joblib==1.0.1
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lightdm-gtk-greeter-settings==1.2.2
lxml==4.6.3
MarkupSafe==1.1.1
menulibre==2.2.1
mugshot==0.4.2
murmurhash==1.0.5
netifaces==0.10.4
numpy==1.20.2
oauthlib==3.1.0
olefile==0.46
onboard==1.4.1
packaging==20.9
pandas==1.2.4
parsel==1.6.0
pathy==0.5.2
pexpect==4.6.0
Pillow==7.0.0
preshed==3.0.5
priority==1.3.0
Protego==0.1.16
psutil==5.5.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycparser==2.20
pycups==1.9.73
pydantic==1.7.3
PyDispatcher==2.0.5
PyGObject==3.36.0
PyJWT==1.7.1
pymacaroons==0.13.0
pymongo==3.11.3
PyNaCl==1.3.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
python-apt==2.0.0+ubuntu0.20.4.5
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
pytz==2021.1
PyYAML==5.3.1
queuelib==1.6.1
reportlab==3.5.34
requests==2.22.0
requests-unixsocket==0.2.0
scikit-learn==0.24.1
scipy==1.6.3
Scrapy==2.5.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
service-identity==18.1.0
sgt-launcher==0.2.5
simplejson==3.16.0
six==1.14.0
sklearn==0.0
smart-open==3.0.0
soupsieve==2.2.1
spacy==3.0.6
spacy-legacy==3.0.5
srsly==2.4.1
systemd-python==234
thinc==8.0.3
threadpoolctl==2.1.0
tqdm==4.60.0
Twisted==21.2.0
typer==0.3.2
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.8
w3lib==1.22.0
wadllib==1.3.3
wasabi==0.8.2
xcffib==0.8.1
xkit==0.0.0
zope.interface==5.4.0

I receive the error ERROR: Could not find a version that satisfies the requirement for the following packages:

apturl==0.5.2
catfish==1.4.13
command-not-found==0.3
cupshelpers==1.0
defer==1.0.6
distro-info===0.23ubuntu1
language-selector==0.1
lightdm-gtk-greeter-settings==1.2.2
menulibre==2.2.1
mugshot==0.4.2
onboard==1.4.1
PyGObject==3.36.0
python-apt==2.0.0+ubuntu0.20.4.5
python-debian===0.1.36ubuntu1
screen-resolution-extra==0.0.0
sgt-launcher==0.2.5
systemd-python==234
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
xkit==0.0.0

(I tried to eliminate them one by one until the container could be compiled, but of course it did not run because of the absence of these packages).

I also tried to replace pypy by the normal python image, but I received the same error.

I tried to use the following dockerfile based on an ubuntu image :

FROM ubuntu:focal

SHELL ["/bin/bash", "-xe", "-c"]

ARG DEBIAN_FRONTEND=noninteractive

COPY . /code

ADD requirements.txt ./

RUN apt-get update -q \
 && apt-get install -y -q --no-install-recommends \
        python3-wheel \
        python3-pip \
        gunicorn \
 && if [ -e requirements.txt ]; then \
        python3 -m pip install --no-cache-dir \
            --disable-pip-version-check \
            -r requirements.txt; \
    fi \
 && python3 -m pip install \
        --no-cache-dir --disable-pip-version-check \
        /code/ \
 && apt-get remove -y python3-pip python3-wheel \
 && apt-get autoremove -y \
 && apt-get clean -y \
 && rm -rf /var/lib/apt/lists/* \
 && useradd _gunicorn --no-create-home --user-group

USER _gunicorn
WORKDIR /code

CMD ["gunicorn", \
     "--bind", "0.0.0.0:8000", \
     "hello_world:app"]

I also got the same result.

I tried to edit the docker DNS options by :

• Adding DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" to the file /etc/default/docker.

• Adding { "dns": ["192.168.1.254", "8.8.8.8"] } to the file /etc/docker/daemon.json.

I feel that I ran out of propositions :(. Does anyone have an idea of what can I do to make pip install all these packages in a python image container?

Thanks.

Answer 1

I think David Maze nailed it in the comments for some of the failures: a lot of the Python packages in the fail-list are installed via apt together with the Ubuntu packages they are part of. If you look them up (e.g. distro-info, ufw, command-not-found) on https://packages.ubuntu.com/, you'll see that many apt package ships with a Python library. (In fact, python-apt seems to be an outlier for being in PyPI as most of them are missing completely, and this answer starts with explaining why.)

The above only applies to packages that fail with the following exact message:

  ERROR: Could not find a version that satisfies the 
         requirement <package-name> (from versions: none)
  ERROR: No matching distribution found for <package-name>

Found it important to emphasize this because in my case (see at the bottom) this was the issue, and once I simply omitted the "Ubuntu-shipped packages", I just had to resolve other dependency issues, such as PyGObject (which was a tough one).

Also, there are other variants of the above error message (e.g., here and here), and this Stackoverflow thread, Could not find a version that satisfies the requirement <package>, has a lot of suggestions for those cases.

---

My case

Inherited a legacy Django app, and the dev did a pip freeze before leaving the project 2 years ago. I simply removed all the qualifiers (i.e., == + parts until the end of the line), and started the pip install process when I got these errors.

I have a theory that needs to be confirmed, but it was true in my case:

The pip freeze was issued on the production server, and it caught every Python package that could be pulled into the project (regardless whether it was used by the project or not).

So kind of like capturing the output of pip list vs. pip list --local.

I also had ufw on my list (among many others) that didn't make any sense as it is a firewall and this project was a simple, internal CMS. From then on, I kept hammering pip install, and crossed out errored out packages which were there because of (probably) unrelated Ubuntu installs.

---

Sidenote: Python modules with external dependencies

For example, PyGObject and co. I lucked out using Nix, especially when it comes to declaring external dependencies. Used nix-shell first to resolve dependency issues to find the right dependencies,

nix-shell -p cairo gobject-introspection

which found the needed headers automatically. Later, I added into buildInputs when specifying a Nix shell expression. Even the database was dropped in a nix-shell.

^{The Django project itself is here.}