How can I generate a requirements.txt file for a package not available on my development platform?

Question

I'm trying to generate requirements/dev.txt and prod.txt files for my python project. I'm using pip-compile-multi to generate them from base.in dev.in and prod.in files. Everything works great until I add tensorflow-gpu==2.0.0a0 into the prod.in file. I get this error when I do: RuntimeError: Failed to pip-compile requirements/prod.in.

I believe this is because tensorflow-gpu is only available on Linux, and my dev machine is a Mac. (If I run pip install tensorflow-gpu==2.0.0a0 I am told there is no distribution for my platform.) Is there a way I can generate a requirements.txt file for pip for a package that is not available on my platform? To be clear, my goal is to generate a requirements.txt file using something like pip-compile-multi (because that will version dependencies) that will only install on Linux, but I want to be able to actually generate the file on any platform.

Answer 1

Use environment markers from PEP 496:

tensorflow-gpu==2.0.0a0; sys_platform!='darwin'

Answer 2

You could run pip-compile-multi in a Docker container. That way you'd be running it under Linux, and you could do that on your Mac or other dev machines. As a one-liner, it might look something like this:

docker run --rm --mount type=bind,src=$(pwd),dst=/code -w /code python:3.8 bash -c "pip install pip-compile-multi && pip-compile-multi"

I haven't used pip-compile-multi, so I'm not exactly sure how you call it. Maybe you'd need to add some arguments to that command. Depending on how complicated your setup is, you could consider writing a Dockerfile and simplifying the one-liner a bit.