Robust Config Loading for Python Apps

Posted on Wed 27 October 2021 in Development • 3 min read

At work, this week, we've been updating our python client, which allows customers to programmatically interact with our platform.

While the basic configuration of the client is quite simple (requiring just access credentials), however there are more than a dozen configuration options that can be overridden for development purposes, but can't easily be set at runtime without access to the underlying codebase. Although trivial for a developer to change, when running automated testing or handing off to QA, this becomes a bit of an issue.

In order to simplify these use cases both for our internal engineering team, and possibly for end users with a more complex configuration, I wanted to add a mechanism to the app that would allow configuration to be specified in multiple ways, depending on whatever was easiest for the particular user's use-case.

The Goal

I want application config to be taken in the following order:

  1. Parameters directly passed to the application
  2. Per-setting environmental variables (e.g. MYAPP_SOMEVAR=abcd)
  3. A user-defined config file, passed with a specific envvar (MYAPP_CONFIG=~/my-config.ini)
  4. A config file loaded from a known-location, relative to the user's home directory (~/.myapp/config.ini)
  5. The bundled default app config.

Anything up higher in the list will take precedence over config specified in one of the lower methods in the list.

Background Information

Before we really get started, we're going to take a brief look at our app's starting point.

Project Structure

Before we get in to some of the further details, we'll briefly look at how the application is structured, as this will have some importance, later, when looking at the setuptools configuration files.

Our Starting Point

The client itself is a pretty simple class, that a user can instantiate with some values (they're unimportant for this post, but in our app it's just an application key and secret, which are used for authentication).

# src/innotescus/innotescus.py

class ApiClient:
    def __init__(api_key: str, api_secret: str):
        # .. initialize authorization and store token ..

    def foo():
        # make api call and parse response data
        return bar

# expected usage
from innotescus.innotescus import ApiClient

my_client = ApiClient(
    api_key='some-key',
    api_secret='some-secret'
)
my_client.foo()

As stated above, the major problem wish this is that we've several configuration options that need set when working with the library in development mode or when testing against staging/development servers.

[Step 1] Bundling the Default App Config

Looking at the previous image, src/innotescus/config.ini will contain our application's DEFAULT configuration values (the ones that will be read if a value is not specified by the user).

[DEFAULT]
server_url = innotescus.app
port = 443
auth_domain = auth.innotescus.app
audience = https://innotescus-prod.auth0.com/api/v2/
scope =
ssl_verification = Yes

force_insecure_channel = No
client_id =
client_secret =

[innotescus]

In order to load this config, we'll be using python's configparser package in the standard lib.

There are two important things to note here: 1. We've included the

Now, since this will be distributed through pypi, we'll need to make sure our config file is included with our distribution.

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.7
include_package_data = True

[options.packages.find]
where = src

[options.package_data]
* = *.ini

[Step 2] Supporting User-Defined Configuration Files

[Step 3] Supporting Environmental Configuration

Wrapping Up - The Complete Client Factory

def client_factory(client_id=None, client_secret=None) -> ApiClient:
    """ Builds and configures a new `ApiClient`.

    :param `str` client_id: API credentials (create from web admin)
    :param `str` client_secret: API credentials (create from web admin)
    :return: A configured api client, ready for use
    """
    cp = ConfigParser()
    _loaded_paths = cp.read([
        os.path.abspath(os.path.join(os.path.dirname(__file__), 'config.ini')),
        os.path.expanduser('~/.innotescus/client.ini'),
        os.getenv('INNO_CONFIG', ''),
    ])
    env_conf = {
        'server_url': os.getenv('INNO_SERVER_URL'),
        'port': os.getenv('INNO_SERVER_PORT'),
        'auth_domain': os.getenv('INNO_AUTH_DOMAIN'),
        'audience': os.getenv('INNO_AUDIENCE'),
        'scope': os.getenv('INNO_SCOPE'),
        'ssl_verification': os.getenv('INNO_SSL_VERIFICATION'),
        'force_insecure_channel': os.getenv('INNO_FORCE_INSECURE_CHANNEL'),
        'client_id': client_id or os.getenv('INNO_CLIENT_ID'),
        'client_secret': client_secret or os.getenv('INNO_CLIENT_SECRET'),
    }
    env_conf = {k: v for k, v in env_conf.items() if v}  # strip null env vars

    cp.read_dict({'innotescus': env_conf})

    inno_config = cp['innotescus']

    auth_client = InnoAuthClient(
        client_id=inno_config.get('client_id'),
        client_secret=inno_config.get('client_secret'),
        scope=inno_config.get('scope'),
        audience=inno_config.get('audience'),
        auth_domain=inno_config.get('auth_domain')
    )

    return InnoApiClient(
        channel_factory_=partial(channel_factory, inno_config=inno_config, auth_client=auth_client),
        auth_client=auth_client,
        api_base_url=f'https://{inno_config.get("server_url")}:{inno_config.get("port")}',
        verify_ssl=inno_config.getboolean('ssl_verification')
    )

python