Python - setuptools - working on two dependent packages (in a single repo?)

Question

I am working on a python project which contains two modules. The modules are very closely related and hence I want them to be in the same git repo, and to be able to develop them together in my IDE:

• module1 depends on module2
• module1 has lots of other heavy dependencies, module2 does not
• module1 and module2 will be used in different runtime environments
• module2 should be installable separately so it can run in e.g. an AWS Lambda

Consequently I have tried to set up a project structure which contains the two modules in two folders within one repo, each folder having a setup.py so it can packaged. Is this a reasonable approach?

\module1
  setup.py
  \module1
    __init__.py
    [scripts].py
\module2
  setup.py
  \module2
    __init__.py
    [scripts].py

The above structure should allow me to work on the project locally treating module2 as a regular module, but the setup.py file means that it can be distributed as it's own package, right?

The problem is that I can't include the module2 dependency in module1's setup.py:

from setuptools import setup

setup(
    name="module1",
    version="0.1",
    packages=['module1'], # I really need to include module2 scripts here, right?...
    install_requires=['pandas', 'numpy', ...]
)

Does anyone have any advice on how to approach this problem? The two solutions I can guess are:

• Packaging and publishing module2 on it's own before depending on it from module1. This makes development much more infleixble
• Nesting module2 within module1 so I can include it in the packages argument to the setup(...) function. This breaks the clarity that module1 should be treating module2 as if it were essentially an external dependency...

Answer 1

Your main reason to keep them in the same repository is to enable parallel work on both the modules in your IDE. I would suggest to keep the modules in separate repositories and use the editable mode for pip or develop mode for setuptools to achieve the "parallel" development.

Essentially, you would be installing module2 in a develop/editable mode when you use it in module1. Thereafter, you will see all the changes made to module2 immediately while developing module1

Doing the same will also make sure that things like pip install git+<git_directory> do not break and you can install directly from git