StatBuffer

DataBuffer is an efficient way to maintain a local buffer list associated with a given key that can later be flushed to persistent storage. In fast moving systems, this provides a scalable way keep track of data without putting heavy loads on a database.

Installation

The package can be installed by adding data_buffer to your list of dependencies in mix.exs:

def deps do
  [
    {:data_buffer, "~> 0.1"}
  ]
end

Documentation

Please see HexDocs for additional documentation.

Getting Started

We can start off by creating our buffer. This is simply a module that uses DataBuffer and implements the handle_flush/2 as well as the optional handle_error/2 callback.

defmodule Buffer do
  use DataBuffer

  @impl DataBuffer
  def handle_flush(key, data) do
    # do database stuff...

    # we must return an :ok atom
    :ok
  end
end

We then must add the buffer to our supervision tree.

children = [
  Buffer
]

There are some configruable options available for our buffers. You can read more about them here. These options can be passed when creating our buffer.

use DataBuffer, partitions: 4, interval: 10_000

With our buffer started, we can now insert data. A key can be any valid term.

Buffer.insert("mykey", "myval1") # Adds myval1 to mykey
Buffer.insert("mykey", "myval2") # Adds myval2 to mykey

Our buffer will be flushed using our handle_flush/2 callback after the default interval period. Dead counters are automatically removed.

def handle_flush(key, data) do
  IO.inspect(key) # returns "mykey"
  IO.inspect(data) # returns ["myval1", myval2"]
  :ok
end

If the handle_flush/2 returns an invalid value or raises an exception, the flush operation will be retried after the configurable :retry_delay up to the maximum :retry_max. If the operation still fails, the handle_error/2 callback will be called. It is then left up to the developer how to handle the data.

def handle_error(key, data) do
  # Put the data back into the buffer...
  # Or put the data to local disk...
  :ok
end