pip always reinstalls zipped packages even if already installed

Question

I am building a requirements.txt file to deploy a set of Python packages across my team via pip. I have saved the packages themselves on a local server to minimize internet downloads, so I have a bunch of wheels and gz/zip files in a folder.

Example test.txt:

packages\networkx-2.3.zip
packages\cx_Oracle-7.1.3-cp37-cp37m-win_amd64.whl
packages\SQLAlchemy-1.3.3.tar.gz

For whl files, pip correctly skips already installed packages, but for the gz/zip ones, it always reinstalls them.

Here is what I get when I run my test.txt file through pip:

λ pip install -r test.txt
Processing c:\users\xxxx\dev\python deploy\packages\networkx-2.3.zip
Processing c:\users\xxxx\dev\python deploy\packages\sqlalchemy-1.3.3.tar.gz
Requirement already satisfied: cx-Oracle==7.1.3 from file:///C:/Users/xxxx/dev/Python%20deploy/packages/cx_Oracle-7.1.3-cp37-cp37m-win_amd64.whl in c:\users\xxxx\appdata\local\programs\python\python37\lib\site-packages (from -r test.txt (line 2)) (7.1.3)
Requirement already satisfied: decorator>=4.3.0 in c:\users\xxxx\appdata\local\programs\python\python37\lib\site-packages (from networkx==2.3->-r test.txt (line 1)) (4.4.0)
Installing collected packages: networkx, SQLAlchemy
  Found existing installation: networkx 2.3
    Uninstalling networkx-2.3:
      Successfully uninstalled networkx-2.3
  Running setup.py install for networkx ... done
  Found existing installation: SQLAlchemy 1.3.3
    Uninstalling SQLAlchemy-1.3.3:
      Successfully uninstalled SQLAlchemy-1.3.3
  Running setup.py install for SQLAlchemy ... done
Successfully installed SQLAlchemy-1.3.3 networkx-2.3

How can I skip reinstalling zipped packages?

Answer 1

This behaviour is expected because pip can't compare the wheel file hash against the installed package. Either use dependency specifications combined with find-links instead of direct paths to files in test.txt:

--find-links=packages
networkx==2.3
SQLAlchemy==1.3.3

Or convert source dists to wheels with

$ pip wheel -r test.txt --wheel-dir packages

and use resulting wheels instead of source dists in test.txt:

packages\networkx-2.3-py2.py3-none-any.whl
packages\SQLAlchemy-1.3.3-cp36-cp36m-win_amd64.whl

This should also speed up the installation time since wheels are already prebuilt. However, packages with C extensions (like SQLAlchemy in this case) will produce wheels that are not platform- or python-agnostic, so e.g. SQLAlchemy-1.3.3-cp36-cp36m-win_amd64.whl can only be installed for Python 3.6 on a 64-bit windows. If your team members have different working environments (e.g. some use WIndows, some use MacOS etc), you would need to provide prebuilt wheels for each environment; in this case, stick with the find-links solution.