Releasing DynamoDB Config Store

I am releasing DynamoDB Config Store open source today. The project provides easy configuration management for Python projects. The configuration options are stored in AWS DynamoDB and can be created, updated and fetched using the DynamoDBConfigStore class.

Storing configuration in DynamoDB has some advantages, for example:

• The configuration is easily and consistently available across many nodes
• One option can consist of many keys and values

Imaging a DynamoDB table like this:

------------+----------------+----------------+----------------+----------------
_store*     | _option**      | host           | port           | secret-key
------------+----------------+----------------+----------------+----------------
prod        | db             | db-cluster.com | 27017          |
prod        | external-port  |                | 80             |
prod        | secret-key     |                |                | abcd1234
test        | db             | localhost      | 27017          |
test        | external-port  |                | 4000           |
prod        | secret-key     |                |                | test1234
------------+----------------+----------------+----------------+----------------

*) Hash key
**) Range key

The table contains two Stores (prod and test) and each Store has three Options. Take a closer look at the db option. It contains both a host key and a port key.

DynamoDB Config Store supports a few different ways to retrieve configuration. Currently we support:

• Simple config store (SimpleConfigStore)
• Time based config store (TimeBasedConfigStore)

The Simple config store will always query DynamoDB for the latest configuration option. This behavior will consume more reads, but in return you’ll always get the latest configuration back.

The Time based config store read data from DynamoDB on a schedule and exposes all options as instance attributes, e.g. store.config.option. That approach reduces the read unit consumption, but is not “strongly” consistent as all configuration updates are not reflect until the next config reload.

I will go through the details of the SimpleConfigStore below.

Installation

The easiest way to install DynamoDB Config Store is via pip:

pip install dynamodb-config-store

Instanciate a Store

A store can be instanciated like this:

from dynamodb_config_store import DynamoDBConfigStore

store = DynamoDBConfigStore(
    connection,                 # boto.dynamodb2.layer1.DynamoDBConnection
    table_name,                 # Table name to use
    store_name)                 # Store name to use

When the Store is instanciated it will create the table if it does not exist. Currently a table with 1 read unit and 1 write unit will be created.

If the Store is instanciated towards an existing table, it will check that the table schema matches the expected Store schema. In other words it will check that store_key is the table hash key and that option_key is the table range key.

Write configuration

You can insert new items in the configuration via the set method.

# 'option' is the name of the option where you want to store the values
store.set('option', {'key1': 'value', 'key2': 'value'})

set takes an option (str) and a dict with keys and values.

Reading configuration

Read a specific Option

Specific Options can be fetched like this:

store.config.get('option')

You will get a dict with all Keys related to the given Option:

{
    'key1': 'value1',
    'key2': 'value2',
    'key3': 'value3',
    'key4': 'value4'
}

Get specific Keys from an Option

If you don’t need all Keys in the response, you can select which Keys you want to retrieve by adding the keys parameter:

store.config.get('option', keys=['key1', 'key4'])

Returns:

{
    'key1': 'value1',
    'key4': 'value4'
}

Fetching all Options in a Store

If you want to retrieve all Options within a Stor, simply omit the option in the request:

store.config.get()

You will get an dictionary like this in return:

{
    'option1': {
        'key1': 'value1',
        'key2': 'value2'
    },
    'option2': {
        'key1': 'value1',
        'key4': 'value2'
    }
}

Wrapping up

Thanks for reading this long post :). Let me know if you find any issues with the project, and don’t forget to star it on GitHub!.