Handling Configuration
One of the things you may have heard about deploying releases is that
configuration is hard. This was never exactly completely true, but in
previous iterations of the release tooling, it was certainly more difficult to
deal with some edge cases; particularly dealing with applications that abuse the
application env for configuration, rather than using parameterized processes and
functions. This pain was due to the fact that Mix’s config file, config.exs
,
was not usable directly in releases.
Background
Releases instead required a sys.config
file, essentially a static form of
configuration defined in Erlang syntax. In order to build releases for Elixir
applications, older tooling compiled the results of evaluating the Mix config
file to sys.config
format, and bundled that into the release.
This meant that some constructs for runtime configuration, namely dynamic
things, like System.get_env/1
would be evaluated at build time, not runtime,
which was a source of confusion. Older tools introduced REPLACE_OS_VARS
, an
environment variable you export at runtime which instructed the tool to replace
occurances of ${VAR}
with the value from the system environment in the
sys.config
file.
The problem with REPLACE_OS_VARS
is that it’s essentially a hack, and as such
had limitations - you could only use the ${VAR}
syntax in strings, because
otherwise Mix couldn’t evaluate the config. The result is that you could only do
“dynamic” configuration with string values, and not values of other types.
This was generally only a problem with applications that were not well behaved with regard to configuration - but sometimes you can’t choose your dependencies, and so a solution was still needed.
Included Applications
One approach to working around the problem of configuring an application needing
dynamic configuration at boot is adding it to :included_applications
.
Applications in this list are not automatically started by the runtime, instead
it is up to your application to start them as part of your supervision tree.
Doing so allows you to perform some work before that application is started,
namely configuration.
The downside of :included_applications
is that an application can only be
included once, and any applications which depend on it being started must also
be included, or they will never start and your application will never start,
leading to a chicken and egg problem where you can’t start the included
application because your app can’t start because the included application isn’t started!
In general this approach should be reserved for when it is the only solution, or when your application truly needs to control the lifecycle of the included application. Usually this only applies to applications within your stack, not dependencies.
Alternative Tools
There were a variety of tools which were born out of the need to work around
runtime configuration of poorly designed applications. One of which I wrote,
called Conform. Libraries like this
worked as release plugins, manipulating the sys.config
file via hooks at
runtime to inject custom configuration.
While such tools work, they are inherently fragile, as they needed to do
orchestration via shell scripts to properly handle replacing configs without
destroying originals, playing nice with upgrades, and integrating with some of
the release tooling’s implicit behavior. Ideally we wanted something that was
integrated at the same level as the sys.config
mechanism itself.
Config Providers
In 2.0+, Distillery provides a framework for loading configuration at runtime
from sources other than sys.config
, called Config Providers. Providers are
just modules which implement a behavior, and are configured via
rel/config.exs
. You can have no config providers (using just sys.config
) or
many providers (running in order and applying configuration on top of the last).
Config providers are executed during boot after all applications are loaded, but
before any applications have been started - with the exception of required
applications kernel
, stdlib
, compiler
, and elixir
. This means config
providers are operating in a highly restrive environment, where many things may
not be available. However, they are executing early enough in the boot process
to configure any application except those mentioned above (of which only
kernel
has associated configuration parameters, most of which are static or
not typical for many applications, we will detail the approach to configuring
kernel
below).
Config providers can be used by adding the provider module and it’s
configuration to rel/config.exs
like so:
environment :prod do
set config_providers: [
{Toml.Provider, [path: "${RELEASE_ROOT_DIR}/config.toml"]}
]
set overlays: [
{:copy, "config/defaults.toml", "config.toml.example"}
]
end
The above (using the provider from
toml) instructs Distillery to modify
the boot script for the release to execute the provider with
Toml.Provider.init([path: "..."])
at the appropriate point during boot. If you
are wondering how to get a config file in the release, the above example also
shows how you might use an overlay to copy a default config file into the
release, to a location which will not overwrite a previous config when the
release is extracted, but will be handy when configuring the release for the
first time.
Any provider module must implement the Mix.Releases.Config.Provider
behavior.
See it’s documentation for details.
This configuration framework supports a variety of interesting configuration
scenarios, such as reading from files like .toml
, .yaml
, or .json
; using
:httpc
to request configuration from a metadata endpoint like etcd or Consul.
Mix Config Provider
Out of the box, Distillery also includes a provider for Mix.Config
, which
allows you to use your config.exs
files with a release. You have to opt in to
this provider, because Mix.Config
blends compile-time and runtime
configuration options into one file, and many legacy configs make assumptions
about the environment the config is evaluated in (namely that a Mix project is
available, or that commands like git
can be invoked and have meaningful
output). You can use it like so:
environment :prod do
set config_providers: [
{Mix.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/var/config.exs"]}
]
end
The path given here is where Distillery will place the config.exs
which is
built for the release (it combines config.exs
and any of it’s imported files
for the environment the release is built in, to a single file).
Kernel Configuration
To configure the kernel
application, you have a few options:
Option 1: erl_opts
One approach is to provide configuration via :erl_opts
:
environment :prod do
set erl_opts: "-kernel sync_nodes_mandatory '[foo@hostname, bar@hostname]'"
end
This works well for simple config values, but is unwieldy for complex values (such as the example above).
Option 2: vm.args
Another approach is to set config flags via vm.args
:
environment :prod do
set vm_args: "rel/prod.vm.args"
end
# prod.vm.args
-kernel sync_nodes_mandatory '[foo@${HOSTNAME}, bar@${HOSTNAME}]'
This approach is a much better way of handling arguments to the VM during boot.
And with REPLACE_OS_VARS=true
exported in the runtime environment, values can
even be dynamic, as shown in the example above, where ${HOSTNAME}
would be
replaced with the value of the system HOSTNAME
environment variable.
The downside is that it still relies on some degree of static information, e.g.
the nodes in the sync_nodes_mandatory
list above are known in advance in the
example, but what about when the node names are not known?
Option 3: sys.config
When Distillery builds a release, a sys.config
file is still generated, and
contains values from config.exs
. You can configure :kernel
there, and the
settings will come through. Unfortunately this is not much better than the
vm.args
solution, as you are limited by information known when the release is built.
Option 4: Runtime Generation
If none of the above are workable solutions, you are likely in a situation where
you will need to construct vm.args
or sys.config
dynamically as part of your
deployment, or using boot hooks like pre_configure
. You would do something
like the following:
environment :prod do
# The template for vm.args
set vm_args: "rel/prod.vm.args"
# The hook to mutate the template
set pre_configure_hooks: ["rel/hooks/pre_configure.d/generate_vm_args.sh"]
end
# prod.vm.args
-kernel sync_nodes_mandatory '[${SYNC_NODES_MANDATORY}]'
#!/usr/bin/env bash
# hooks/pre_configure.d/generate_vm_args.sh
export SYNC_NODES_MANDATORY
if [ -z "$HOSTNAME" ]; then
echo 'Expected $HOSTNAME to be set in the environment, unable to configure mandatory nodes!'
exit 1
fi
# If no nodes defined, set it to an empty list
if [ -z "$NODES"]; then
SYNC_NODES_MANDATORY=""
else
# Expects $NODES to be something like 'foo,bar,baz'
for node in `echo "$NODES" | sed -e s/,/\n/g`; do
if [ -z "$SYNC_NODES_MANDATORY" ]; then
# If no nodes defined yet, adding the first
SYNC_NODES_MANDATORY="\\'${node}@${HOSTNAME}\\'"
else
# Otherwise append an element to the list with ,
SYNC_NODES_MANDATORY="${SYNC_NODES_MANDATORY},\\'${node}@${HOSTNAME}\\'"
fi
done
fi
This setup above will generate, at runtime, a final vm.args
that looks
something like the following, given NODES="foo,bar,baz"
and HOSTNAME=host.local
:
-kernel sync_nodes_mandatory '[\'foo@host.local\',\'bar@host.local\',\'baz@host.local\']'
Option 5: Configuration Management
Last, but not least, using configuration management systems like Salt, Puppet,
Chef, Ansible, etc., are great ways to handle generating the vm.args
or other
configuration files during deployment. They typically have better templating
tools so that you don’t have to write something from scratch in shell.
If this is an option for you, I would start here before reaching for any of the above options.
Wrap up
As you can see, configuring kernel
is not particuarly easy, compared to the
methods for configuring the rest of the system; luckily this is a special case
scenario, which most applications may never encounter. But if you do have a
need, I hope the above gives you a good lay of the land, and some options for
working around the discomfort.
Configuring Applications
Lastly, I want to provide some general guidance on designing your applications to be easily configurable when used as a component in a larger system.
Rule 1: Parameters, not environment
If you need some piece of configuration, and you are wondering how best to
handle getting it to where it is needed, always try to pass configuration as
parameters first, rather than using the application environment (e.g.
Application.get_env/2
).
Non-OTP Applications
For non-OTP applications, this is simple, have all your API functions which require configuration, accept a list of options as a parameter, and pass it down the stack to where it’s needed.
You can make this parameter default to an empty list, or some predetermined set
of defaults with opts \\ []
or opts \\ @defaults
, like so:
defmodule Log do
@default_device :standard_io
@default_opts [device: @default_device]
def info(msg, opts \\ @default_opts) do
device = Keyword.get(opts, :device, @default_device)
IO.puts(device, msg)
end
end
In the example above, users of this API can use Log.info(msg)
for the most
part, unless they need to change the defaults. In cases where you can’t accept a
default, require the extra parameter, or use Keyword.fetch!/2
to raise an
error if an option was not provided.
OTP Applications
For OTP applications, parameterization starts at the root of your application,
in the application callback module, namely in the start/2
callback:
defmodule MyApp do
use Application
def start(_type, _args) do
Supervisor.start_link(__MODULE__, [], name: __MODULE__.Supervisor)
end
def init(_) do
all = Application.get_all_env(:myapp)
config = [
port: get_in(all, [__MODULE__.Component, :port]) || System.get_env("PORT"),
# ...
]
children = [
{__MODULE__.Component, [config]},
# ...
]
Supervisor.init(children, strategy: :one_for_one)
end
end
In the above example, our application will fetch configuration from the application environment, but then build up the actual configuration which will be passed to children of the root supervisor by first looking for a value in the application env, then falling back to other methods. Once constructed, this configuration is passed down the supervision tree, from parent to child, starting at the root.
This approach can be extracted into a MyApp.Config
module, where you can put
all the logic for fetching configuration and converting it into it’s final form,
keeping your application callback module clean.
An additional benefit of this approach is that in your tests, you can start many
copies of __MODULE__.Component
with different configuration, allowing you to
test that component in parallel. This requires taking an approach to designing
your processes so that they are not relying on singletons or other global state,
but the very first step is taking configuration as parameters.
“A la carte” OTP Applications
This approach to designing applications is similar to what I just described, but
your application callback module does not actually start any children. Instead,
users of your application start things like __MODULE__.Component
within their
own supervision tree, and take over responsibility for providing configuration
via parameters.
You may still have need for some global state, or singleton processes, which do not require configuration, and are invisible to users of your application. In these cases your application callback module will start only these children, and leave the rest up to users of the application. You can see this approach in action in the Postgrex library, for example.
run_erl
The run_erl
command is responsible for running a release on Unix systems,
capturing STDERR and STDOUT so that all output can be logged as well as
allowing monitoring and remote debugging of a running release.
Several environment variables are useful in configuring run_erl
, for
example to customize logging output. To specify environment variables to
apply to run_erl
, you can add a line like
set run_erl_env: "RUN_ERL_LOG_MAXSIZE=10000000 RUN_ERL_LOG_GENERATIONS=10"
in your release configuration.
This configuration can also be specified in the RUN_ERL_ENV
environment variable at the time of running mix release
.
For a complete list of environment variables respected by run_erl
, see
here.