Structure python project with multiple entry points and centralised (config) files

Question

I'm writing a bunch of python scripts that all work together, for instance tool1.py loads some data into a database table, tool2.py reads from this table, does some calculations and writes the results into another table, and daemon1.py is a webserver serving the result. Each tool and daemon has a whole bunch of support files (that are needed only for that one tool), and needs its own directory. In addition, I do have some code shared between all tools, such as config.py and database.py. Intuitively, I structured the project thus:

/README.md
/requirements.txt
/config.py # contains shared configuration
/database.py # shared code to connect to the database
/tool1/
    tool1.py # entrypoint
    [...] # bunch of other files only needed by tool1
/tool2/
    tool2.py #entrypoint
    [...] # bunch of other files only needed by tool2
/deamon1/
    daemon1.py #entrypoint
    [...] # bunch of other files only needed by daemon1

I then run my tools and daemons with the command python tool1/tool1.py. The problem however here is how tool1.py has access to config.py/database.py. I considered the following options, would be interested in what is considered the "right" way in python, or any alternatives I might have missed (perhaps a different way to lay out the project). Extra karma will be rewarded for a link to an authoritative answer.

1. symlink the config.py/database.py files into the subdirectories. Don't like it much since it confuses my editor, and seems to make things more complicated than necessary.
2. make config.py/database.py in some separate package, which I install in my virtualenv. Don't like it since I'm constantly changing these files as well, and I want to keep them in the same git repo.
3. change the sys.path at the top of tool1.py. This results in 4 lines at the top of each file, plus importing of the sys and os modules for no other reason than to set these items.

import os
import sys
sys.path.append(os.path.join(os.path.abspath(
    os.path.dirname(__file__)), ".."))

4. Add the toplevel path to $PYTHONPATH
5. create toplevel entrypoints for tool1.py/tool2.py/daemon1.py that read something like (after renaming the tool1 directory to tool1dir)

from tool1dir import tool1
tool1.run()

6. Put config.py/database.py into a separate package and symlink that directory from each subdir.

As noted, would like to hear the pythonesque way to do this, or any suggestions or preferences.

Answer 1

Recently I revisited this issue, and looked at how I would solve it in 2024. I wrote up my answer on my blog, however I will copy down the main items here:

• Make the whole thing into a proper python project:

/README.md
/pyproject.toml
/my_project/
  config.py # contains shared configuration
  database.py # shared code to connect to the database
  tool1/
    tool1.py # entrypoint
    [...] # bunch of other files only needed by tool1
  tool2/
    tool2.py #entrypoint
    [...] # bunch of other files only needed by tool2
  deamon1/
    daemon1.py #entrypoint
    [...] # bunch of other files only needed by daemon1

The pyproject.toml may be minimal:

[project]
name = "my_project"
version = "0.1.0"
dependencies = [
    ...
]

Now the my_project/tool1/tool1.py file can use from .. import database but only if it's called as a module.

In order to call it as a module, install it: pip install . (or pip install -e . in order to install it in dev mode). Now tool1 can be called as python -m my_project.tool1.tool1.

You can even go one step further, defining your entrypoints in pyproject.toml:

...

[project.scripts]
tool1 = "my_project.tool1.tool1:main"

This will (after reinstalling the package) allow just typing tool1 on the commandline, and it will run the main() function in my_project/tool1/tool1.py.