View Source DetsPlus (DetsPlus v2.1.5)

DetsPlus persistent tuple/struct/map storage.

DetsPlus has a similiar API as dets but without the 2GB file storage limit. Writes are buffered in an internal ETS table and synced every auto_save period to the persistent storage.

While sync() or auto_save are in progress the database is still readable and writeable.

There is no commitlog so not synced writes are lost. Lookups are possible by key and non-matches are accelerated using a bloom filter. The persistent file concept follows DJ Bernsteins CDB database format, but uses an Elixir encoded header https://cr.yp.to/cdb.html. When syncing a new CDB database file is created and replaces the old CDB atomically file using File.rename! so database corruptions are not possible from incomplete updates.

Limits are:

  • Total file size: 18_446 Petabyte
  • Maximum entry size: 4 Gigabyte
  • Maximum entry count: :infinity

Example:

{:ok, dets} = DetsPlus.open_file(:example)
DetsPlus.insert(dets, {1, 1, 1})
[{1, 1, 1}] = DetsPlus.lookup(dets, 1)
:ok =  DetsPlus.close(dets)

Summary

Functions

Returns a specification to start this module under a supervisor.

Syncs pending writes to the persistent file and closes the table.

Returns the number of object in the table. This is an estimate and the same as info(dets, :size).

Deletes all objects with key Key from table Name.

Deletes all objects from a table in almost constant time.

Deletes all instances of a specified object from a table.

Returns information about table Name as a list of objects

Returns the information associated with item for the table. The following items are allowed

Inserts one or more objects into the table. If there already exists an object with a key matching the key of some of the given objects, the old object will be replaced.

Inserts one or more objects into the table. If there already exists an object with a key matching the key of some of the given objects, the old object will be replaced.

Returns a list of all objects with key Key stored in the table.

Works like lookup/2, but does not return the objects. Returns true if one or more table elements has the key key, otherwise false.

Opens an existing table or creates a new table. If no file argument is provided the table name will be used. Dets registers a Process under the provided name which can be used for calling alternatively to the pid.

Reducer function following the Enum protocol.

Starts a sync of all changes to the disk. Same as sync/1 but doesn't block

Ensures that all updates made to table are written to disk. While the sync is running the table can still be used for reads and writes, but writes issued after the sync/1 call will not be part of the persistent file. These new changes will only be included in the next sync call.

Types

@type t() :: %DetsPlus{
  ets: term(),
  hashfun: term(),
  keyfun: term(),
  keyhashfun: term(),
  pid: pid()
}

Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

@spec close(t() | pid() | atom()) :: :ok

Syncs pending writes to the persistent file and closes the table.

@spec count(t() | pid() | atom()) :: integer()

Returns the number of object in the table. This is an estimate and the same as info(dets, :size).

@spec delete(t() | pid() | atom(), any()) :: :ok | {:error, atom()}

Deletes all objects with key Key from table Name.

@spec delete_all_objects(t() | pid() | atom()) :: :ok | {:error, atom()}

Deletes all objects from a table in almost constant time.

Link to this function

delete_object(pid, object)

View Source
@spec delete_object(t() | pid() | atom(), tuple() | map()) :: :ok | {:error, atom()}

Deletes all instances of a specified object from a table.

@spec info(t() | pid() | atom()) :: [] | nil

Returns information about table Name as a list of objects:

  • {file_size, integer() >= 0}} - The file size, in bytes.
  • {filename, file:name()} - The name of the file where objects are stored.
  • {keypos, keypos()} - The key position.
  • {size, integer() >= 0} - The number of objects estimated in the table.
  • {type, type()} - The table type.
@spec info(
  t(),
  :file_size
  | :header_size
  | :filename
  | :keypos
  | :size
  | :type
  | :creation_stats
  | :bloom_bytes
  | :hashtable_bytes
) :: any()

Returns the information associated with item for the table. The following items are allowed:

  • {file_size, integer() >= 0}} - The file size, in bytes.
  • {header_size, integer() >= 0}} - The size of erlang term encoded header.
  • {bloom_bytes, integer() >= 0}} - The size of the in-memory and on-disk bloom filter, in bytes.
  • {hashtable_bytes, integer() >= 0}} - The size of the on-disk lookup hashtable, in bytes.
  • {filename, file:name()} - The name of the file where objects are stored.
  • {keypos, keypos()} - The key position.
  • {size, integer() >= 0} - The number of objects estimated in the table.
  • {type, type()} - The table type.
@spec insert(t() | pid() | atom(), tuple() | map() | [tuple() | map()]) ::
  :ok | {:error, atom()}

Inserts one or more objects into the table. If there already exists an object with a key matching the key of some of the given objects, the old object will be replaced.

@spec insert_new(t() | pid() | atom(), tuple() | map() | [tuple() | map()]) ::
  true | false

Inserts one or more objects into the table. If there already exists an object with a key matching the key of some of the given objects, the old object will be replaced.

@spec lookup(t() | pid() | atom(), any()) :: [tuple() | map()] | {:error, atom()}

Returns a list of all objects with key Key stored in the table.

Example:

2> State.open_file(:abc)
{ok,:abc}
3> State.insert(:abc, {1,2,3})
ok
4> State.insert(:abc, {1,3,4})
ok
5> State.lookup(:abc, 1).
[{1,3,4}]

If the table type is set, the function returns either the empty list or a list with one object, as there cannot be more than one object with a given key. If the table type is bag or duplicate_bag, the function returns a list of arbitrary length.

Notice that the order of objects returned is unspecified. In particular, the order in which objects were inserted is not reflected.

@spec member(t() | pid() | atom(), any()) :: false | true | {:error, atom()}

Same as member?/2

@spec member?(t() | pid() | atom(), any()) :: false | true | {:error, atom()}

Works like lookup/2, but does not return the objects. Returns true if one or more table elements has the key key, otherwise false.

Link to this function

open_file(name, args \\ [])

View Source

Opens an existing table or creates a new table. If no file argument is provided the table name will be used. Dets registers a Process under the provided name which can be used for calling alternatively to the pid.

Arguments:

  • file - An optional path + filename for the database file.
  • auto_save - The autosave interval. If the interval is an integer Time, the table is flushed to disk whenever it is not accessed for Time milliseconds. If the interval is the atom infinity, autosave is disabled. Defaults to 180_000 (3 minutes).
  • auto_save_memory - The autosave threshold in memory. When the internal ETS table reaches a size bigger than this the table is flushed to disk. Defaults to 1_000_000_000 (1 GB)
  • page_cache_memory - The amount of memory to use for file system caching. Defaults to 1_000_000_000 (1 GB)
  • keypos - The position of the element of each object to be used as key. Defaults to 1. The ability to explicitly state the key position is most convenient when we want to store Erlang records in which the first position of the record is the name of the record type.
@spec reduce(t() | pid() | atom(), any(), (... -> any())) :: any()

Reducer function following the Enum protocol.

@spec start_sync(t() | pid() | atom()) :: :ok

Starts a sync of all changes to the disk. Same as sync/1 but doesn't block

@spec sync(t() | pid() | atom()) :: :ok

Ensures that all updates made to table are written to disk. While the sync is running the table can still be used for reads and writes, but writes issued after the sync/1 call will not be part of the persistent file. These new changes will only be included in the next sync call.