Computational distributable flows with stages.

Inspired by Flow, Spark, and Amazon’s EMR. Elastic Flow provides a conceptual structure for distributing an enumerable data source to a cluster of servers and then aggregate into a result as data completes processing.

With Elastic Flow you create a Flow based program as normal; you’d then set your module in Elastics config along with your master/slave setup. Work will then be distributed from master to your cluster to each server which are all running the same program on each server, as a single BEAM app. As work is completed the results are sent back to master to be aggregated. There are default aggregation methods but the common usage will be to define your own aggregate method. By defining an intercept module, actions can be viewed as they happen.

This is currently in the working proof of concept stage. It fulfilled the author’s (my) use case and as tested more will be expanded. There is included an example that can be run from within the library locally to simulate a distributed system, which is a great way to get a feel for the BEAM and Elastic Flow.

Flow: https://github.com/elixir-lang/flow/

Example

Let’s run the example first! We can use the library on it’s on before creating a full app. A word counter will be run, just like the example used in Flow’s docs. Estimated time: 5 minutes.

Step 1: Clone the repository from github https://github.com/Tyler-pierce/ElasticFlow.git

  terminal 1> mix deps.get

Step 2: Change the options in config/config.exs to use your computer name for the servers. The rest of the options are defaulted to use the word counting app.

  config :elastic_flow, 
    servers: %{
      :"yam1@your-computer-name" => :master, 
      :"yam2@your-computer-name" => :slave, 
      :"yam3@your-computer-name" => :slave
    }

Step 3: Now make sure you have 3 console windows/tabs open, and you can start your 3 ‘servers’

  terminal 1> iex --sname yam1 --cookie yamrider -S mix run

  terminal 2> iex --sname yam2 --cookie yamrider -S mix run

  terminal 3> iex --sname yam3 --cookie yamrider -S mix run

Step 4: Run the convenience function to connect the servers (in production you’d have your cluster setup by your deployment or vm.args) and then run the program.

  iex(yam1@your-computer-name)1> ElasticFlow.Example.setup()

  iex(yam1@your-computer-name)1> ElasticFlow.Example.run()

# You should see a lot of output because of the example apps custom interceptor. Check the other tabs and you should see yam2 and yam3 have received and sent data (and left receipts behind). Feel free to take the interceptor out of the config to avoid the noise.

Step 5: Check your results!

  iex(yam1@your-computer-name)1> ElasticFlow.Example.results()

    %{"first" => 7, "whole" => 14, ... }

And that’s it. If you wish to explore further try turning on the chaos_spirits option (from 1 to 45) to force errors, or adding longer jobs to try step queueing. And have a look through the example folder code to see how you’d structure your app. I’ll continue to update it as features progress so each integration point is obvious.

Cheers!

Github README: https://github.com/Tyler-pierce/ElasticFlow