Including Package Data Python from Top-level when Package is in Subdirectory

Question

I'm trying to package a series of configuration files along with some source code. I have a directory structure like this (which I cannot change, due to the nature of the team)

.
├── configs
│   ├── machines
│   ├── scope
├── esm_tools
│   ├── __init__.py
├── README.rst
├── setup.cfg
├── setup.py

61 directories, 45 files (Truncated)

From what I understand here (https://docs.python.org/3/distutils/setupscript.html#installing-package-data), I can add some parts to the setup call:

setup(
    # ... other stuff
    include_package_data=True,
    name="esm_tools",
    packages=["configs", "esm_tools"],
    package_dir={"configs": "configs", "esm_tools": "esm_tools"},
    package_data={'configs': ['*']},
    version="4.1.5",
    zip_safe=False,
)

However, I can't access the package data with:

In [1]: import pkg_resources                                                                                                                                                                                                                                                                       

In [2]: pkg_resources.resource_listdir("esm_tools", "config")                                                                                                                                                                                                                                      
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-2-f0f255c14df6> in <module>
----> 1 pkg_resources.resource_listdir("esm_tools", "config")

/global/AWIsoft/miniconda/4.7.12/lib/python3.7/site-packages/pkg_resources/__init__.py in resource_listdir(self, package_or_requirement, resource_name)
   1161         """List the contents of the named resource directory"""
   1162         return get_provider(package_or_requirement).resource_listdir(
-> 1163             resource_name
   1164         )
   1165 

/global/AWIsoft/miniconda/4.7.12/lib/python3.7/site-packages/pkg_resources/__init__.py in resource_listdir(self, resource_name)
   1439 
   1440     def resource_listdir(self, resource_name):
-> 1441         return self._listdir(self._fn(self.module_path, resource_name))
   1442 
   1443     def metadata_listdir(self, name):

/global/AWIsoft/miniconda/4.7.12/lib/python3.7/site-packages/pkg_resources/__init__.py in _listdir(self, path)
   1608 
   1609     def _listdir(self, path):
-> 1610         return os.listdir(path)
   1611 
   1612     def get_resource_stream(self, manager, resource_name):

FileNotFoundError: [Errno 2] No such file or directory: '/home/ollie/pgierz/dev/esm_tools/esm_tools/esm_tools/config'

Any help would be greatly appreciated, I'm not sure what I'm doing wrong...

Answer 1

Based on your call pkg_resources.resource_listdir("esm_tools", "config"), I assume you want to remap configs to esm_tools.config in the installed package:

site-packages
├── esm_tools
│   ├── __init__.py
│   ├── config
│   │   ├── machines
│   │   ├── scope

This means you have to do the following things:

1. Tell setuptools to include a subpackage esm_tools.config (even if it doesn't really exist in source code base and is technically a namespace one, we'll construct it via further configuration)
2. Tell setuptools where the sources for the new esm_tools.config package are located (this is again just a necessary measure to tell setuptools what to include in the dist. The package itself won't provide any Python sources since no Python files are located in configs).
3. Tell setuptools to include package data for esm_tools.config from the correct path.

Example:

setup(
    ...,
    packages=['esm_tools', 'esm_tools.config'],           # 1
    package_dir={'esm_tools.config': 'configs'},          # 2
    package_data={'esm_tools.config': ['../configs/*']},  # 3
)

Note that this won't work with editable installs (neither via pip install --editable . nor with python setup.py develop), so you will have to construct more or less ugly local workarounds with symlinks or .pth files or whatever. The wheel dist (built via python setup.py bdist_wheel or pip wheel) will work out of the box, for source dists you'll have to include the configs dir via MANIFEST.in as package_data won't be read at sdist time:

# MANIFEST.in
...
graft configs