Fact.EventIndexer behaviour (Fact v0.1.0)

View Source

Base behaviour and macro for building event indexers.

Fact.EventIndexer defines the callbacks and GenServer scaffolding shared by all event indexers in the Fact storage system. An indexer listens for new events and extracts zero, one, or many values from each even it processes. For each extracted value, the indexer create or appends to a file, recording the event's Fact.record_id/0.

Indexers are just event projections, that filter the event ledger, and produce keyed, ordered sets of events.

Safe to Delete

It's safe to delete any of index files and folders written to the file system, when the system is not operating. They will be recreated, the next time indexer is started.

Behaviour

An indexer must implement the index_event/3 callback, which extracts values from the supplied event.

The __using__/1 macro injects the full GenServer implementation that:

  • initializes and ensures the index exists
  • rebuilds the index from history on startup
  • subscribes to live event notifications
  • updates the index as new events arrive

Custom Indexers

Custom indexers only need to implement the callback.

Examples

This would produce an index for every user, including all the events which define a user_id in the event data.

defmodule YourApp.UserIndexer do
  use Fact.EventIndexer

  @impl true
  def index_event(%{@event_data => %{"user_id" => user_id}}, _opts), 
    do: to_string(user_id)

  def index_event(_, _), do: nil
end

This would produce an index for every tenant, including all the events which define a tenant_id in the event metadata.

defmodule YourApp.TenantIndexer do
  use Fact.EventIndexer
  def index_event(%{@event_metadata => %{"tenant_id" => tenant_id}}, _opts), 
    do: to_string(tenant_id)

  def index_event(_, _), do: nil        
end

Summary

Types

The values that can be return by a c:Fact.EventIndexer.index_event callback function.

This describes the results of the indexing process.

The value produced by an Fact.EventIndexer.indexer_id/0 when indexing an Fact.event_record/0.

The message that is published immediately after an indexer processes a Fact.record/0.

Custom option values passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of records.

The unique identifier for an indexer.

This is additional metadata for a specific Fact.EventIndexer.indexer_id/0.

A module that implements the Fact.EventIndexer behaviour to index records.

Option values passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of of records.

Options passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of records.

Option values used by the start_link/2 functions for indexer modules.

Options used by the start_link/2 functions for indexer modules.

t()

The state structure used by indexers in the GenServer callback functions.

Callbacks

Called when an event needs to be indexed.

Functions

Subscribe to messages published by the specified indexer.

Gets the name of the topic where the indexer publishes messages.

Types

index_event_result()

@type index_event_result() :: index_value() | [index_value()] | nil

The values that can be return by a c:Fact.EventIndexer.index_event callback function.

index_result()

@type index_result() :: %{
  position: Fact.event_position(),
  record_id: Fact.record_id(),
  index_values: [index_value()]
}

This describes the results of the indexing process.

index_value()

@type index_value() :: String.t()

The value produced by an Fact.EventIndexer.indexer_id/0 when indexing an Fact.event_record/0.

indexed_message()

@type indexed_message() :: {:indexed, indexer_id(), index_result()}

The message that is published immediately after an indexer processes a Fact.record/0.

indexer_custom_option()

@type indexer_custom_option() :: {atom(), term()}

Custom option values passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of records.

indexer_id()

@type indexer_id() :: indexer_module() | {indexer_module(), indexer_key()}

The unique identifier for an indexer.

Built-in Indexers

indexer_key()

@type indexer_key() :: String.t()

This is additional metadata for a specific Fact.EventIndexer.indexer_id/0.

At the time of writing, only Fact.EventDataIndexer uses an Fact.EventIndexer.indexer_key/0, because there can be multiple processes running, each indexing a different key within an Fact.event_data/0

indexer_module()

@type indexer_module() :: :atom

A module that implements the Fact.EventIndexer behaviour to index records.

indexer_option()

@type indexer_option() :: {:indexer_key, indexer_key()} | indexer_custom_option()

Option values passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of of records.

indexer_options()

@type indexer_options() :: [indexer_option()]

Options passed to the Fact.EventIndexer.index_event/3 callback function to control the indexing of records.

start_option()

@type start_option() ::
  {:indexer_id, indexer_id()}
  | {:indexer_opts, indexer_options()}
  | GenServer.option()

Option values used by the start_link/2 functions for indexer modules.

start_options()

@type start_options() :: [start_option()]

Options used by the start_link/2 functions for indexer modules.

t()

@type t() :: %{
  database_id: Fact.database_id(),
  indexer: indexer_id(),
  indexer_opts: indexer_options(),
  checkpoint: Fact.event_position(),
  schema: Fact.event_record_schema()
}

The state structure used by indexers in the GenServer callback functions.

Callbacks

index_event(schema, event, indexer_options)

@callback index_event(
  schema :: Fact.event_record_schema(),
  event :: Fact.event(),
  indexer_options()
) :: index_event_result()

Called when an event needs to be indexed.

Functions

subscribe(database_id, indexer)

@spec subscribe(Fact.database_id(), indexer_id()) :: :ok

Subscribe to messages published by the specified indexer.

Messages

topic(indexer)

@spec topic(indexer_id()) :: String.t()

Gets the name of the topic where the indexer publishes messages.