Mifrat (mifrat_ex v0.3.1)

(Mifrat is an acronym of Many Indexed Fields Random Access Table)

Module to manage an in-memory table with primary_key and secondary indexes. The objective is to have a way to store complex temporary records with fast access through any indexed field.

To manage data you can use a classic functional style:

get(:users, 1234)
get(:users, :name, "Benito Camela")
delete(:users, 5432)
insert(:users, ...)
# etc...

or you can use a pseudo query language:

use Mifrat

# Get data
mifrat_query from: :users, get: {id, name}
mifrat_query from: :users, get: {id, name}, when: id >= 10 and id <= 15
mifrat_query from: :users, get: {id, name, dni}, when: year, in_range: {1940,1950}

# Delete data
mifrat_query from: :users, delete_when: id < 5
mifrat_query from: :users, delete: 5 # delete record by primary_key
mifrat_query from: :users, delete: [1, 2, 3, 4] # delete records by primary_keys
mifrat_query from: :users, delete_when_in_range: {1,4} # delete records by a range of primary_keys
mifrat_query from: :users, delete_when: year, in_range: {1940, 1950}

# Add/update data
# insert_into will push new data if the primary_key does not exists, otherwise update fields
mifrat_query insert_into: :users, values: {21, "Cachirulo Gonzalez", 1995, 2599945445, 1, 22845856}

NOTE: In the when expressions you can use ONLY guard enable functions.

TODO: a more complex expression parser that allow like, ilike, lower, upper, and many more inline functions to filter records.

Installation

def deps do
  [
    {:mifrat_ex, "~> 0.3.1"}
  ]
end

Summary

Functions

Return the number of records of the table.

Return the count of records that match with the pattern and/or guard. The pattern and the guard are strings. If pattern is :full it will be expanded to a tuple of all fields taking the field names declared with new/2.

Delete a record by primary_key.

Delete one or more records by a secondary_key.

WARNING!!! This function remove physically every record in the table.

Delete many records referenced with its primary key.

Delete a bunch of records referenced by a range of its primary key.

Delete a bunch of records referenced by a range of one of its secondary indexes keys.

Delete the table and his indexes. No data survive this process. If the table has enable autosave, before delete the data will be flushed to disk (see new/3).

Return the full record referenced by the primary_key.

Return one o more records using a secondary index key. Return the the auxiliary indexed record or the full main record of the table. Option :return can be

Delete a bunch of records referenced by a range of one of its primary key or its secondary indexes keys. If you have a table like this

Insert or update a record. If the primary key does not exists it insert, otherwise update. The fields values must follow the order declared with new/2 or new/3. It can be a list or a tuple.

Load the table from file in pathname.

Convert a map into a tuple record.

Do it easy, make queries with a more natural language.

Create a table. The fields are defined with a keyword list; each pair has a field name (the key) and an index type (the value). The order of the fields is important and the primary key must be the first.

Convert a tuple record into a map. If the first argument is not a tuple, return just the argument passed. That is because many times the record come from a get/n call and can take as value :not_found.

Clear every secondary indexes and rebuild every one again from zero.

Flush the table to disk in the file pointed by pathname.

Types

field_range()

@type field_range() :: {any(), any()}

Functions

count(table)

@spec count(table :: atom() | :ets.tid()) :: integer()

Return the number of records of the table.

custom_count(table, pattern \\ :full, guard \\ "true")

@spec custom_count(
  table :: atom() | :ets.tid(),
  pattern :: :full | String.t(),
  guard :: String.t()
) ::
  integer()

Return the count of records that match with the pattern and/or guard. The pattern and the guard are strings. If pattern is :full it will be expanded to a tuple of all fields taking the field names declared with new/2.

# Example
custom_count(:users, "{_id, _name, year, _phone}", "year == 1985")

delete(table, primary_key)

@spec delete(table :: atom() | :ets.tid(), primary_key :: any()) :: :not_found | :ok

Delete a record by primary_key.

delete(table, field_name, key)

@spec delete(table :: atom() | :ets.tid(), field_name :: atom(), key :: any()) ::
  integer()

Delete one or more records by a secondary_key.

delete_all(table)

@spec delete_all(table :: atom() | :ets.tid()) :: :ok | :error

WARNING!!! This function remove physically every record in the table.

delete_list(table, list)

@spec delete_list(table :: atom() | :ets.tid(), pk_list :: list()) :: :ok

Delete many records referenced with its primary key.

delete_range(table, from, to)

@spec delete_range(table :: atom() | :ets.tid(), from :: any(), to :: any()) :: :ok

Delete a bunch of records referenced by a range of its primary key.

delete_range(table, field_name, from, to)

@spec delete_range(
  table :: atom() | :ets.tid(),
  field_name :: atom(),
  from :: any(),
  to :: any()
) ::
  integer()

Delete a bunch of records referenced by a range of one of its secondary indexes keys.

destroy(table)

@spec destroy(atom() | :ets.tid()) :: :ok | :error

Delete the table and his indexes. No data survive this process. If the table has enable autosave, before delete the data will be flushed to disk (see new/3).

get(table, primary_key)

@spec get(table :: atom() | :ets.tid(), primary_key :: any()) :: :not_found | tuple()

Return the full record referenced by the primary_key.

get(table, field_name, key, opts \\ %{return: :records})

@spec get(
  table :: atom() | :ets.tid(),
  field_name :: atom(),
  key :: any(),
  options :: map() | list()
) ::
  [tuple()]

Return one o more records using a secondary index key. Return the the auxiliary indexed record or the full main record of the table. Option :return can be:

  • return: :records: return a list of full records from the table
  • return: :keys: return a list of auxiliary secondary index record ({{:<index>, key}, primary_key})

get_range(table, field_name, from, to, opts \\ %{return: :keys, limit: :infinity})

Delete a bunch of records referenced by a range of one of its primary key or its secondary indexes keys. If you have a table like this:

new(:users, [
  id: :primary_key
  name: :unindexed,
  year: :indexed_non_uniq,
  phone: :indexed
], [
  ...
])

You can get a range of records using the primary key:

get_range(table, :id, 100, 200)  # get the records with id >= 100 and <= 200

or get a range of records using the key of a secondary index:

get_range(table, :year, 1940, 1950)  # get the records with year >= 1940 and <= 1950

Just as in get/3 you can get a list of full records from the table or a list of auxiliary secondary index record (return: :records or return: :keys)

imft_query(params)

(macro)

insert(table, record)

@spec insert(table :: atom() | :ets.tid(), list() | tuple()) ::
  :duplicate_record | [:skip | true | false]

Insert or update a record. If the primary key does not exists it insert, otherwise update. The fields values must follow the order declared with new/2 or new/3. It can be a list or a tuple.

insert(:users, {1, "Jorge Luis Borges", 1899, 542915040798})

or

insert(:users, [1, "Jorge Luis Borges", 1899, 542915040798])

list_table_indexes(table, type \\ [:indexed, :indexed_non_uniq])

load(pathname)

@spec load(pathname :: String.t()) ::
  {:ok, table :: atom() | :ets.tid()} | {:error, term()}

Load the table from file in pathname.

map_to_record(map, table)

@spec map_to_record(map :: map(), table :: atom() | :ets.tid()) :: map()

Convert a map into a tuple record.

mifrat_query(list)

(macro)
@spec mifrat_query(from: table :: atom() | :ets.tid(), get: tuple(), when: any()) :: [
  tuple()
]
@spec mifrat_query(from: table :: atom() | :ets.tid(), get: tuple()) :: [tuple()]
@spec mifrat_query(
  from: table :: atom() | :ets.tid(),
  get: tuple(),
  when: any(),
  in_range: field_range()
) :: [tuple()]
@spec mifrat_query(from: table :: atom() | :ets.tid(), delete_when: any()) :: [
  tuple()
]
@spec mifrat_query(from: table :: atom() | :ets.tid(), delete: [term()]) :: :ok
@spec mifrat_query(from: table :: atom() | :ets.tid(), delete: term()) :: :ok
@spec mifrat_query(
  from: table :: atom() | :ets.tid(),
  delete_when: atom(),
  in_range: field_range()
) ::
  integer()
@spec mifrat_query(
  from: table :: atom() | :ets.tid(),
  delete_when_in_range: field_range()
) :: integer()
@spec mifrat_query(insert_into: table :: atom() | :ets.tid(), values: tuple()) :: [
  true | false | :skip
]

Do it easy, make queries with a more natural language.

This macro define a pseudo query languate to get/delete/insert data in the table. If you have a table :users with this struct:

[
  id: :primary_key,
  name: :unindexed,
  year: :indexed_non_uniq,
  phone: :indexed,
  category: :indexed_non_uniq,
  dni: :indexed
]

You could do:

# Get data
mifrat_query from: :users, get: {id, name}
mifrat_query from: :users, get: {id, name}, when: id >= 10 and id <= 15
mifrat_query from: :users, get: {id, name, dni}, when: year, in_range: {1940,1950}

# Delete data
mifrat_query from: :users, delete_when: id < 5
mifrat_query from: :users, delete: 5 # delete record by primary_key
mifrat_query from: :users, delete: [1, 2, 3, 4] # delete records by primary_keys
mifrat_query from: :users, delete_when_in_range: {1,4} # delete records by a range of primary_keys
mifrat_query from: :users, delete_when: year, in_range: {1940, 1950}

# Add/update data
# insert_into will push new data if the primary_key does not exists, otherwise update fields
mifrat_query insert_into: :users, values: {21, "Cachirulo Gonzalez", 1995, 2599945445, 1, 22845856}

new(table_name, fields, options \\ [])

@spec new(name :: atom(), fields :: keyword(), opts :: keyword()) :: :ets.tid()

Create a table. The fields are defined with a keyword list; each pair has a field name (the key) and an index type (the value). The order of the fields is important and the primary key must be the first.

new(:users, [
  id: :primary_key
  name: :unindexed,
  year: :indexed_non_uniq,
  phone: :indexed
], [
  ...opts...
])

The fields type available are:

  • :primary_key: just one field can has this type and it is the main index of the table.
  • :indexed: the field will has an auxiliary uniq index.
  • :indexed_non_uniq: the field will has an auxiliary non uniq index.
  • :unindexed: the field won't be indexed, it is just data.

The options available are:

  • :_autosave: Force the flush of the table to disk every :period (see below) in :path file (see below). If true make mandatory :path. Default false.
  • :period: Set how often autosave will flush to disk. It is a value in miliseconds. Default 300_000 (5 minutes).
  • :path: The path filename where the table will be flushed. Ignored if autosave: false.
  • :initial_load: If true and autosave: true will try to load from :path the table. Ignored if autosave: false. Default is true.

record_to_map(record, table)

@spec record_to_map(record :: tuple(), table :: atom() | :ets.tid()) :: map()

Convert a tuple record into a map. If the first argument is not a tuple, return just the argument passed. That is because many times the record come from a get/n call and can take as value :not_found.

reindex(table)

Clear every secondary indexes and rebuild every one again from zero.

store(table, pathname)

@spec store(table :: atom() | :ets.tid(), pathname :: String.t()) :: true | false

Flush the table to disk in the file pointed by pathname.