I am releasing DynamoDB Config Store open source today. The project provides easy configuration management for Python projects. The configuration options are stored in AWS DynamoDB and can be created, updated and fetched using the DynamoDBConfigStore
class.
Storing configuration in DynamoDB has some advantages, for example:
- The configuration is easily and consistently available across many nodes
- One option can consist of many keys and values
Imaging a DynamoDB table like this:
------------+----------------+----------------+----------------+----------------
_store* | _option** | host | port | secret-key
------------+----------------+----------------+----------------+----------------
prod | db | db-cluster.com | 27017 |
prod | external-port | | 80 |
prod | secret-key | | | abcd1234
test | db | localhost | 27017 |
test | external-port | | 4000 |
prod | secret-key | | | test1234
------------+----------------+----------------+----------------+----------------
*) Hash key
**) Range key
The table contains two Stores (prod
and test
) and each Store has three Options. Take a closer look at the db
option. It contains both a host
key and a port
key.
DynamoDB Config Store supports a few different ways to retrieve configuration. Currently we support:
- Simple config store (
SimpleConfigStore
)
- Time based config store (
TimeBasedConfigStore
)
The Simple config store will always query DynamoDB for the latest configuration option. This behavior will consume more reads, but in return you’ll always get the latest configuration back.
The Time based config store read data from DynamoDB on a schedule and exposes all options as instance attributes, e.g. store.config.option
. That approach reduces the read unit consumption, but is not “strongly” consistent as all configuration updates are not reflect until the next config reload.
I will go through the details of the SimpleConfigStore
below.
Installation
The easiest way to install DynamoDB Config Store is via pip
:
pip install dynamodb-config-store
Instanciate a Store
A store can be instanciated like this:
from dynamodb_config_store import DynamoDBConfigStore
store = DynamoDBConfigStore(
connection, # boto.dynamodb2.layer1.DynamoDBConnection
table_name, # Table name to use
store_name) # Store name to use
When the Store is instanciated it will create the table if it does not exist. Currently a table with 1 read unit and 1 write unit will be created.
If the Store is instanciated towards an existing table, it will check that the table schema matches the expected Store schema. In other words it will check that store_key
is the table hash key and that option_key
is the table range key.
Write configuration
You can insert new items in the configuration via the set
method.
# 'option' is the name of the option where you want to store the values
store.set('option', {'key1': 'value', 'key2': 'value'})
set
takes an option
(str
) and a dict
with keys and values.
Reading configuration
Read a specific Option
Specific Options can be fetched like this:
store.config.get('option')
You will get a dict
with all Keys related to the given Option:
{
'key1': 'value1',
'key2': 'value2',
'key3': 'value3',
'key4': 'value4'
}
Get specific Keys from an Option
If you don’t need all Keys in the response, you can select which Keys you want to retrieve by adding the keys
parameter:
store.config.get('option', keys=['key1', 'key4'])
Returns:
{
'key1': 'value1',
'key4': 'value4'
}
Fetching all Options in a Store
If you want to retrieve all Options within a Stor, simply omit the option
in the request:
store.config.get()
You will get an dictionary like this in return:
{
'option1': {
'key1': 'value1',
'key2': 'value2'
},
'option2': {
'key1': 'value1',
'key4': 'value2'
}
}
Wrapping up
Thanks for reading this long post :). Let me know if you find any issues with the project, and don’t forget to star it on GitHub!.