Avoiding `sys.path.append(..)` for imports

Question

This isn't the first time I am cringing over imports in Python. But I guess this one is an interesting use case, so I thought to ask it here to get a much better insight. The structure of my project is as follows:

sample_project
   - src
        - __init__.py
        - module1
           - __init__.py
           -  utils.py
        - module2
           - __init__.py 
           - models.py
        - app.py

The module1 imports methods from module2 and app imports method from all the other. Also, when you run the app it needs to create a folder called logs outside of src folder. There are now to ways to run the app:

1. From inside src folder flask run app
2. From outside of src folder flask run src.app

To make sure that I don't get import errors because of the change of the top level module where the app is started, I do this:

import sys
sys.path.append("..")

Is there any better solution to this problem?

Answer 1

The pythonic solution for the import problem is to not add sys.path (or indirectly PYTHONPATH) hacks to any file that could potentially serve as top-level script (incl. unit tests), since this is what makes your code base difficult to change and maintain. Assume you have to reorganize your project structure or rename folders.

Instead this is what editable installs are made for. They can be achieved in 2 ways:

1. pip install --editable <path> (requires a basic setup.py)
2. conda develop <path> (requires the conda-build package)

Either way will add a symlink into your site-packages folder and make your local project behave as if it was fully installed while at the same time you can continue editing.

Always remember: KEEP THINGS EASY TO CHANGE

Answer 2

Pythonic Solution: Use packages and modules as they are intended to be used

Ask yourself, why did you want to create a src directory?

I would suggest that more than likely you wanted to follow a convention you knew from another language. (Maybe Java, maybe C, C++, or something else.)

However, if you use Python packages in the way they are intended to be used, there is a far simpler solution.

First lets review a few key points.

• To run the Python file main.py, you run it using the Python interpreter like so: python3 main.py.
• When the Python interpreter starts, it adds the current working directory to its path. (via sys.path)
• It will also add the directory containing the module to be run to its path
• More information can be found in this documentation page: https://docs.python.org/3/tutorial/modules.html
• One issue with that documentation page is it explains how to modify sys.path from within Python code. The issue with that is it gives developers the idea that this is possible and therefore should be used as a solution to import and path problems when it should not.
• The Python interpreter will search the sys.path and PYTHONPATH directories list for modules and packages to resolve when it sees an import statement
• A Python package is a directory with an __init__.py file
• A Python module is just a regular Python file
• The __init__.py file is there to signal to the Python interpreter that it needs to recursively search subdirectories for more Python packages and modules. This is why an __init__.py is usually empty.
• Without an __init__.py the Python interpreter will simply ignore a directory
• This rule is actually an optmization to prevent the interpreter from becoming slow to start up if there are a large number of subdirectories and files to search
• From this we conclude that all local source code should be resolvable from the same directory as the one used to run the target module (main.py)

With that information, you can re-structure your project:

sample_project/
    my_python_package/
        __init__.py
        sub_package_1/
            __init__.py
            utils.py
        sub_package_2/
            __init__.py 
            models.py
    app.py

Run app.py from the directory sample_project: python3 app.py

You can actually go further. If your project becomes very large, it sometimes makes sense to run modules within packages using python3 -m some_package.some_module. Then everything, including app.py becomes a package. I don't think you need this in this particular case, but if you have large numbers of "executable" Python files which are better grouped into a set of directories, then this is the approach to take.

Note that:

• This solution is simple (bordering on trivial, if not necessarily that obvious)
• There is no src directory. Forget about src. This works well in other languages, it doesn't fit into the Python model for how a project should be structured
• You did not need to modify PYTHONPATH
• You did not need to modify sys.path
• You did not need to write extra code to be able to resolve imports
• This solution is easy to understand, it is straight forward and has minimal complexity

---

An experiment to learn about PYTHONPATH and sys.path

You can find out what PYTHONPATH and sys.path are set to with a short experimental code:

$ cd ~
$ mkdir python-path-test
$ touch python-path-test/main.py 

# main.py

import os
import sys

print(f'PYTHONPATH:')
for string in os.environ.get('PYTHONPATH').split(';'):
    print(string)

print(f'sys.path:')
for string in sys.path:
    print(string)

$ export PYTHONPATH=`pwd`
$ python3 python-path-test/main.py
PYTHONPATH:
/home/username
sys.path:
/home/username
/home/username/python-path-test
/usr/lib/python311.zip
/usr/lib/python3.11
/usr/lib/python3.11/lib-dynload
/usr/local/lib/python3.11/dist-packages
/usr/lib/python3/dist-packages
/usr/lib/python3.11/dist-packages

---

Further explanation in regards to other answers

Let me address the issues with the other answers here. All of the answers provided will work, but none of them take the simplest and "most obviously correct" approach.

The reason for this is the "most obviously correct" approach is not that obvious, especially if you come to Python from other languages where things work differently.

Just to say as well - it took me a long time to figure out the solution to the exact same problem which is shown in the question and I only figured out the solution when I went to work for a firm where someone else had figured this out before me.

Also: None of this is really explained on any documentation page anywhere, so it is hardly surprising that most people get it wrong, or do something unneccessarily complex when it isn't needed.

Overview of other solutions:

So far several other solutions have been proposed:

1. Use setuptools and virtual environments to manage what is known as an "editable install".

I don't like this for two reasons: It is more work than is necessary, and you are pretending that some local source code is a PIP package, when it isn't. It just seems like a bizzare thing to do. (This is exactly what I used to do before realizing there is an easier way.)

2. Write Python code to modify the sys.path or PYTHONPATH environment variable

I don't like this because it is a hack:

• The PYTHONPATH environment variable is intended to be used to store the locations of installed packages on your system
• It should be a semi-permenant thing which doesn't change (often)
• This is similarly the case with sys.path
• The other reason modifying PYTHONPATH is bad is because you are embedding (hiding) some code within your project which does unexpected things
• PYTHONPATH should be managed by the Operating System, or at least by the user in a shell
• In my experience, twiddling things which should be managed by your operating system from within code frequently leads to hard to find bugs and hard to understand code

3. Modify PYTHONPATH from a shell

This is better than the above proposal of modifying it from with Python code, but it just isn't necessary, for the reasons I explained above.

---

Appendix:

To give a little further helpful information. Some languages (and corresponding build tools) are designed with maximum flexibility. Others contain built-in rules which constrain how files and folders should be arranged for the build system to work property. These rules are not always explicit or obvious.

cmake is a good example of a build system which offers maximum flexibility. Many projects contain a src directory, under which all the C/C++ code lives. The reason for this is cmake facilitates using explicit and arbitrary paths to configure the build.

On the other hand, the Rust module system is much more constrained. The existence of a directory "creates" a module (or submodule). Cargo and Rust require you to use the filesystem in a constrained way to get the modular structure you want.

Julia is more similar to C++ in that modules are explicit - there is a module keyword, and this is the only way to create a module. It also has include which can take an arbitrary path - although using the Julia build system in an arbitrary way is not recommended, just as it would not be recommended with cmake.

Finally, Python is a bit more tricky. Similarly to Julia, the build system needs to be told how and where modules can be loaded from. It is generally better not to add lots of arbitrary hard-coded paths to the code or build system. Rather, avoiding this and working with what the language offers natively is preferable.

In both the case of Julia and Python, this means that the interpreter/runtime should be able to load your code without adding additional paths.

With this constraint, you will write a much simpler project structure.

To answer some questions in the comments

Here is what one of my Python projects looks like.

python_project_root_directory/
  .vscode/
    settings.json
  .venv/
  lib_something/
    __init__.py
    lib_something_files.py
  the_main_module
    __main__.py
    __init__.py # might not be requried
    main.py # called from __main__.py
  tests/
    some_group/
      test_something.py
      test_another_thing.py
    another_group/
      test_more_things.py
  Dockerfile

Dockerfile

Note: Does not use .venv, because a Docker container is its own isolated environment. You can use a .venv if you want. Change the command to cmd ["./.venv/bin/python3", "-m", "the_main_module"].

from python:3.12-bookworm
... other stuff ...
run pip3 install --no-cache-dir --upgrade -r requirements.txt
cmd ["python3", "-m", "the_main_module"]

If rather than wanting to run a main module, you want to run a python file as "main", change to cmd ["python3", "main.py"].

settings.json

{
    "python.testing.pytestArgs": [
        "tests"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
}