Python module/plugin loading pattern

Sun, Dec 29 2013 - 3 minute read

When writing a new program in python, I often find myself using the same pattern over and over, and wanted to document it here. The idea is that you have some core functionality of the program, and loadable modules to expand the functionality, with only some of the modules being loaded at runtime. These modules are single files inside a modules/ directory, contain a single Module() class (and optionally others depending on what the module needs), and some other functionality that the core of your program will use. This pattern lets you specify which modules to load in a configuration file, and load them up on demand, passing in some configuration to the module.

First, the code to load the modules:

import ConfigParser
import logging
import os
import time

# Read the config file(s)
c = ConfigParser.SafeConfigParser()
c.read(['/etc/myapprc', os.path.expanduser('~/.myapprc')])

# Load the modules
modules = []  # List of module objects
for s in c.sections():
    # We define which modules to load by the fact that there is a section
    # in the file for that module, starting with "mod_". This lets you
    # have other configuration in the file if needed. You can also get the
    # list of modules to load in some other way if desired.
    if s.startswith("mod_"):
        modulename = s[4:]  # Strip off the mod_
        try:
            # Actually import the module
            module = __import__('modules.%s' % modulename, globals(),
                locals(), [modulename])
        except ImportError, e:
            # Print an error if we failed, and carry on, you could exit
            # here if desired. I also tend to use the logging module, but
            # this could just be a print statement or something else if
            # needed.
            logging.error("Unable to load module %s: %s" %
                          (modulename, e))
            continue

        # Instantiate an instance of the module, and pass in the config
        # from the file. I convert all items to a dict for ease of access
        # by the module. You could also pass the raw config parser object
        # if desired.
        module_object = module.Module(dict(c.items(s)))
        # Add the module to the list. This can be more complex than just
        # storing the module object (e.g. adding a name and some config in a
        # dict) if you needed to store more informaiton.
        modules.append(module_object)

# Do something with the modules here. Simple example here to call all module
# run() methods over and over, sleeping 10 seconds in between.
while True:
    for m in modules:
        m.run()
    time.sleep(10)

This gives you a list of module objects in modules, and you can then call methods on those modules (I tend to have a run() method or have the __init__ method set up callbacks for various events depending on the program’s functionality).

A module itself would be a single python file inside modules (e.g. modules/example.py) and look something like this:

class Module(object):


    def __init__(self, config):
        # config is a dictionary containing the relevant section of the
        # configuration file
        self.foo = config.get('foo', 'some_default')
        # more initialization code here

    # ... other module code goes here

You also need to make sure your modules directory has an __init__.py file (it can be blank) so that python knows the directory contains modules.

This pattern can be expanded on in various ways depending on the needs of your program, but for anywhere where you have pluggable functionality, it works well.