Dugan Chen's Homepage

Various Things

Reading Environment Variables in Python. With Metaclasses.

I work in the animation industry. Here, the standard practice is to use environment variables to control the runtime behavior of software. It is common enough, in fact, that powerful open source projects such as Rez have been started to manage it. The idea is that before launching each application, you set the environment variables as needed to get the behavior you want.

This pattern is, of course, is not entirely without precedent. GCC, for example, uses environment variables such as CFLAGS. It’s also standard in BASH scripting. Slackware package build scripts, for example, are BASH scripts that begin with lines like the following:

SET VERSION=${VERSION:-1.3}

What this means is: if VERSION is set in the environment, use it. If not, use the default value of 1.3. You could therefore override the default value when you launch the script:

VERSION=1.4 ./wine.SlackBuild

Which would then build version 1.4 of Wine instead of version 1.3.

Hopefully, you get the idea by now.

As any Python programmer knows, reading environment variables is easy.

import os
print os.environ.get('VERSION')

If our use of environment variables is heavy instead of sparing, then this just isn’t manageable enough. We’ll want a lookup mechanism safer than magic strings, and we want a central repository (yes, a repository for environment variables) that will handle the logic of the default values and overrides.

First, the default environment values are a prime candidate for storing in a YAML configuration file:

SAVE_FOLDER:
  nt: \\server\save
  posix: /save
SERVER: http://production_server

Yes, some values, such as paths to the same network share, have different defaults depending on which platform you’re running on.

We’ll want this YAML to be deserialized into the following Python class, or at least one with the equivalent behavior. Note that the properties show up when you dir() the class (which means static analysis works). Having the keys as properties also means that the risk of typoing them is much reduced.

import os

class Environment(object):

    @property
    def SAVE_FOLDER(self):
        try:
            return os.environ['SAVE_FOLDER']
        except KeyError:
            if os.name == 'nt':
                return r'\\server\save'
            if os.name == 'posix':
                return r'/save'

            raise Exception

    @property
    def SERVER(self):
        try:
            return os.environ['SERVER']
        except KeyError:
            return 'http://production_server'

How do you programatically set properties on a class? You use a metaclass. Assuming that the above YAML file is stored in “default_environment.yaml”, here’s an implementation:

import functools
import os
import yaml

default_environment = {}
    with open('default_environment.yaml') as f:
        default_environment = yaml.load(f)

class EnvironmentMetaClass(type):
    def __new__(cls, name, bases, dct):
        for key, value in default_environment.iteritems():
            getter = functools.partial(
                get_from_environment, key=key,
                default=get_value(value))
            dct[key] = property(getter)
            return super(EnvironmentMetaClass, cls).__new__(
                cls, name, bases, dct)


def get_value(value):
    if type(value) is dict:
        return value[os.name]
    return value


def get_from_environment(self, key, default):
    try:
        return os.environ[key]
    except KeyError:
        return default


class Environment(object):
    __metaclass__ = EnvironmentMetaClass

The Environment repository class is now easy to use.

environment = Environment()
print environment.SAVE_FOLDER
print environment.SERVER

It respects overrides too.

os.environ['SERVER'] = 'http://staging_server'
print environment.SERVER