Grains - Process Data Flow Orchestration

Grains is a framework that uses declarative flow graphs to implement process data flows.

graph LR
  Recipe[Declarative Graph Description]
  Instantiation[Process Instantiation]
  Runtime[Runtime Architecture]

  Recipe --> Runtime
  Instantiation --> Runtime

Motivation

Implementing data flow graphs with processes can be complicated, as the processes are loosely associated with the implementation. This becomes a problem when the number of processes or the graph becomes large. Grains solves this issue by describing the flow graph in a declarative manner before the processes are started. The declaration makes it visible which processes are connected, and which paths information can take. Since the declarations of the flow graphs are separate from implementation, the structure of the graph is reusable. This allows for systems where the abstractions (structures) stay the same, while the implementation changes.

Advantages to an Implicit Data Flow Graph

• Better system overview
• Enhance maintainability and testability
• Increase performance by separating concurrent tasks clearly

Terminology

• Recipe: The declaration of the data flow graph. This is a map from a process to a list of processes which are receiving data from it.
• Grains: Processes that are part of the data flow graph.
• Bread: The runtime after combining the recipe with its grains.

Special Processes

Grains uses special wrapper processes to abstract common functionality into the library. Currently, we support a periodic grain, which pulls data from its predecessor periodically.

Data Flow Example

In example/heat/, a single feedback loop is present as an example for Grains. The task of the example is to hold the temperature within a room at a specified target temperature by using two dedicated heaters. The control system samples the temperature measured at the heaters and activates heating if needed. This is the data flow graph:

graph LR
  subgraph Control System
    State -->|temperature| Sampler
    State -->|temperature| Persistence
    Sampler -->|temperature| UI
    Sampler -->|temperature| Optimizer
  end

  Optimizer -->|command| Heating

  Heating -->|temperature| State

To implement this graph, we define the recipe:

recipe = Grains.new_recipe(:heat,
  %{
    State => [Grains.periodic(Sampler, 1000), Grains.periodic(Persistence, 500)],
    Sampler => [UI, Optimizer],
  })

The recipe contains 5 grains: the system state, a periodic sampler, a periodic persistence process, a UI and an optimizer. State sends information to the sampler and persistence process, while the sampler sends information to the UI and optimizer. We then define how those grains should be started:

grains = Grains.new(
  %{
    State => {Heat.State, [], []},
    Sampler => {Heat.Sampler, [], []},
    Persistence => {Heat.Persistence, [], []},
    Optimizer => {Heat.Optimizer, [], []},
    UI => {Heat.UI, [], []}
  })

This is similar to child specs for supervisors. We map a name used in the recipe to a module with arguments and options. Now we can combine the recipe with the grains to create a running system:

{:ok, bread} = Grains.start_supervised(recipe, grains)

As heaters are not part of the system, they are started outside of this definition:

{:ok, _} = Heating.start_link(%{serial: 1, server: server})

Example run:

mix run -e Heat.run
==> grains
Compiling 7 files (.ex)
Generated grains app
==> heat
Compiling 1 file (.ex)
Setting target temperature to 23.0 degrees
Current average temperature: 19.98333333333335
Current average temperature: 20.166666666666693
Current average temperature: 20.350000000000037
Current average temperature: 20.53333333333338
Current average temperature: 20.716666666666725
Current average temperature: 20.90000000000007
Current average temperature: 21.083333333333414
Current average temperature: 21.266666666666758
Current average temperature: 21.450000000000102
Current average temperature: 21.633333333333447
Current average temperature: 21.81666666666679
Current average temperature: 22.000000000000135

Trivia

What does a vegetarian zombie eat? Grrrraiiiiiinnnns!

License

Copyright 2019 sonnen eServices GmbH

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.