Macros to effortlessly define highly optimized Enum pipelines

overview

Overview

Enumancer provides a defenum/2 macro, which will convert a pipeline of Enum function calls to an optimized tail-recursive function.

defmodule BlazingFast do
  import Enumancer

  defenum sum_squares(numbers) do
    numbers
    |> map(& &1 * &1)
    |> sum()
  end
end

1..10_000_000 |> BlazingFast.sum_squares()  # very fast
1..10_000_000 |> Enum.map(& &1 * &1) |> Enum.sum()  # super slow
1..10_000_000 |> Stream.map(& &1 * &1) |> Enum.sum()  # super slow

There is no need to add Enum., map/2 will be interpreted as Enum.map/2 within defenum/2.

In order to see the actual functions that are being generated, you can just replace defenum/2 by defenum_explain/2 and the code will be printed in the console.

The defenum_explain/2 approach can be useful if you don't want to take the risk of using Enumancer and macros in your production code, but it can inspire the implementation of your optimized recursive functions.

available-functions

Available functions

Most functions taking an Enumerable and returning a list can be used anywhere in the pipeline (e.g. map/2, filter/2, with_index/2...).

On the other hand, functions taking an Enumerable and returning some non-list accumulator (e.g. sum/1, join/2, max/1...) can only be used at the end of the pipeline. There are other cases like sort/1 or reduce/1 which cannot work without the full list and are also limited to the end of the pipeline.

Functions that need to stop without reducing the Enumerable completely, such as take/2 or any?/1, are not available at this point, but might be implemented in the future.

Also, please note that many functions from the Enum module are accepting optional callbacks to add an extra map or filter step. By design, Enumancer does not implement these. For a very simple reason: the available primitives can be combined at will to reproduce them, without any runtime overhead.

See examples below:

replacing-some-composed-functions

Replacing some "composed" functions

• Instead of |> map_join("-", fun), just use |> map(fun) |> join("-")
• Instead of |> map_intersperse(sep, fun), just use |> map(fun) |> intersperse(sep)
• Instead of |> count(&has_valid_foo?/1), just use |> filter(&has_valid_foo?/1) |> count()
• Instead of |> with_index(fn x, i -> foo(x, i) end), just use |> with_index() |> map(fn {x, i} -> foo(x, i) end)
• Instead of |> Map.new(fn x -> {x.k, x.v} end), just use |> map(fn x -> {x.k, x.v} end) |> Map.new()

anywhere-in-the-pipeline

Anywhere in the pipeline

• Enum.map/2
• Enum.filter/2
• Enum.reduce/2
• Enum.reject/2
• Enum.with_index/1
• Enum.with_index/2 (only accepts integer offset)
• Enum.uniq/1
• Enum.uniq_by/2
• Enum.dedup/1
• Enum.dedup_by/2
• Enum.scan/2
• Enum.map_reduce/3 + elem(0) (not plain Enum.map_reduce/3!, see explanation below)

|> map_reduce(acc, fun) by itself returns a tuple and cannot be piped any further.

But |> map_reduce(acc, fun) |> elem(0) can be piped if you only need the mapped list.

only-at-the-end-of-the-pipeline

Only at the end of the pipeline

• Enum.reduce/2
• Enum.reduce/3
• Enum.max/1
• Enum.max/2 (only with a module argument)
• Enum.min/1
• Enum.min/2 (only with a module argument)
• Enum.count/1
• Enum.sum/1
• Enum.product/1
• Enum.reverse/1
• Enum.join/1
• Enum.join/2
• Enum.intersperse/2
• Enum.sort/1
• Enum.sort/2
• Enum.sort_by/2
• Enum.sort_by/3
• Enum.map_reduce/3 (without being followed by |> elem(0))
• Enum.frequencies/1
• Enum.frequencies_by/2
• Enum.group_by/2
• Map.new/1
• MapSet.new/1