Dynamically loading modules at runtime in Python 3

Question

Which is the recommended way to lazy load files in Python 3?

I've built this function by copy-past from Python 2 code

def get_command(self, ctx, cmd_name):
        ns = {}
        fn = os.path.join(cmd_folder, 'cmd_{}.py'.format(cmd_name))

        with open(fn) as f:
            code = compile(f.read(), str(fn), 'exec')
            eval(code, ns, ns)

        return ns['cli']

but I'm not sure if it is the right way. It uses compile and eval

EDIT

After using import like this:

def get_command(self, ctx, cmd_name):
    cmd_mod = import_module('{}.cmd_{}'.format(cmd_folder, cmd_name))

this is the result:

  File "aws_iam_cli/cli.py", line 23, in get_command
    cmd_mod = import_module('{}.cmd_{}'.format(cmd_folder, cmd_name))
  File "/Users/salvatoremazzarino/awsiam/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named '/Users/salvatoremazzarino/awsiam/aws_iam_cli/commands'

EDIT #2:

def get_command(self, ctx, cmd_name):
        mod = import_module('aws_iam_cli.commands.cmd_{}'
                            .format(cmd_name))
        return mod.cli

and the error:

  File "/Users/salvatoremazzarino/awsiam/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'aws_iam_cli'

This is the dir tree:

├── aws_iam_cli
│   ├── __init__.py
│   ├── cli.py
│   ├── commands
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   └── cmd_dump.py
│   └── provider
│       ├── __init__.py
│       ├── policy.py
│       └── role.py

Inside commands I have a module and I call inside that module:

from aws_iam_cli.provider.role import fetch_roles

Answer 1

You can use importlib.import_module to load modules dynamically at run time. So for example:

from importlib import import_module

# load module
pckg_name = 'commands'  # I assume all your commands are in this pckg
cmd_mod = import_module('{}.cmd_{}'.format(pckg_name, cmd_name))

# run command (I'm assuming the module contains a function with the 'cmd_name')
cmd_mod.cmd_name()

I think this kind of approach will be cleaner and safer and should work with what you are doing.