@spec back_up(GenServer.server(), Path.t()) :: :ok | {:error, term()}

Creates a backup of the database into the target directory path

The directory is created upon calling back_up/2, and an error tuple is returned if it already exists.

The function will block until the backup is completed, then return :ok. The backup does not block other readers or writers, and reflects the database state at the time it was started, without any later write.

After the backup completes successfully, it is possible to open it by starting a CubDB process using the target path as its data directory.

Returns a specification to start this module under a supervisor.

The default options listed in Supervisor are used.

@spec clear(GenServer.server()) :: :ok

Deletes all entries, resulting in an empty database.

The deletion is atomic, and is much more performant than deleating each entry manually.

The operation respects all the guarantees of consistency of other concurrent operations. For example, if select2 was called before the call to clear/1 and is running concurrently, the select2 will still see all the entries.

If a compaction is in progress when clear/1 is called, the compaction is halted, and a new one started immediately after. The new compaction should be very fast, as the database is empty as a result of the clear/1 call.

@spec compact(GenServer.server()) :: :ok | {:error, String.t()}

Runs a database compaction.

As write operations are performed on a database, its file grows. Occasionally, a compaction operation can be run to shrink the file to its optimal size. Compaction runs in the background and does not block operations.

Only one compaction operation can run at any time, therefore if this function is called when a compaction is already running, it returns {:error, :pending_compaction}.

When compacting, CubDB will create a new data file, and eventually switch to it and remove the old one as the compaction succeeds. For this reason, during a compaction, there should be enough disk space for a second copy of the database file.

Compaction can create disk contention, so it should not be performed unnecessarily often.

@spec compacting?(GenServer.server()) :: boolean()

Returns true if a compaction operation is currently running, false otherwise.

@spec current_db_file(GenServer.server()) :: String.t()

Returns the path of the current database file.

The current database file will change after a compaction operation.

example

Example

{:ok, db} = CubDB.start_link("some/data/directory")

CubDB.current_db_file(db)
#=> "some/data/directory/0.cub"

@spec data_dir(GenServer.server()) :: String.t()

Returns the path of the data directory, as given when the CubDB process was started.

example

Example

{:ok, db} = CubDB.start_link("some/data/directory")

CubDB.data_dir(db)
#=> "some/data/directory"

@spec delete(GenServer.server(), key()) :: :ok

Deletes the entry associated to key from the database.

If key was not present in the database, nothing is done.

@spec delete_multi(GenServer.server(), [key()]) :: :ok

Deletes multiple entries corresponding to the given keys all at once, atomically.

The keys to be deleted are passed as a list.

@spec dirt_factor(GenServer.server()) :: float()

Returns the dirt factor.

The dirt factor is a number, ranging from 0 to 1, giving an indication about the amount of overhead disk space (or "dirt") that can be cleaned up with a compaction operation. A value of 0 means that there is no overhead, so a compaction would have no benefit. The closer to 1 the dirt factor is, the more can be cleaned up in a compaction operation.

@spec fetch(GenServer.server(), key()) :: {:ok, value()} | :error

Fetches the value for the given key in the database, or returns :error if key is not present.

If the database contains an entry with the given key and value value, it returns {:ok, value}. If key is not found, it returns :error.

@spec file_sync(GenServer.server()) :: :ok

Performs a fsync, forcing to flush all data that might be buffered by the OS to disk.

Calling this function ensures that all writes up to this point are committed to disk, and will be available after a restart.

If CubDB is started with the option auto_file_sync: true, calling this function is not necessary, as every write operation will be automatically flushed to the storage device.

If this function is NOT called, the operative system will control when the file buffer is flushed to the storage device, which leads to better write performance, but might affect durability of recent writes in case of a sudden shutdown.

@spec get(GenServer.server(), key(), value()) :: value()

Gets the value associated to key from the database.

If no value is associated with key, default is returned (which is nil, unless specified otherwise).

@spec get_and_update(GenServer.server(), key(), (value() -> {any(), value()} | :pop)) ::
  any()

Gets the value corresponding to key and updates it, in one atomic transaction.

fun is called with the current value associated to key (or nil if not present), and must return a two element tuple: the result value to be returned, and the new value to be associated to key. fun may also return :pop, in which case the current value is deleted and returned.

Note that in case the value to update returned by fun is the same as the original value, no write is performed to disk.

@spec get_and_update_multi(
  GenServer.server(),
  [key()],
  (%{optional(key()) => value()} ->
     {any(), %{optional(key()) => value()} | nil, [key()] | nil})
) :: any()

Gets and updates or deletes multiple entries in an atomic transaction.

Gets all values associated with keys in keys_to_get, and passes them as a map of %{key => value} entries to fun. If a key is not found, it won't be added to the map passed to fun. Updates the database and returns a result according to the return value of fun.

The function fun should return a tuple of three elements: {return_value, entries_to_put, keys_to_delete}, where return_value is an arbitrary value to be returned, entries_to_put is a map of %{key => value} entries to be written to the database, and keys_to_delete is a list of keys to be deleted.

The read and write operations are executed as an atomic transaction, so they will either all succeed, or all fail. Note that get_and_update_multi/3 blocks other write operations until it completes.

example

Example

Assuming a database of names as keys, and integer monetary balances as values, and we want to transfer 10 units from "Anna" to "Joy", returning their updated balance:

{anna, joy} = CubDB.get_and_update_multi(db, ["Anna", "Joy"], fn entries ->
  anna = Map.get(entries, "Anna", 0)
  joy = Map.get(entries, "Joy", 0)

  if anna < 10, do: raise(RuntimeError, message: "Anna's balance is too low")

  anna = anna - 10
  joy = joy + 10

  {{anna, joy}, %{"Anna" => anna, "Joy" => joy}, []}
end)

Or, if we want to transfer all of the balance from "Anna" to "Joy", deleting "Anna"'s entry, and returning "Joy"'s resulting balance:

joy = CubDB.get_and_update_multi(db, ["Anna", "Joy"], fn entries ->
  anna = Map.get(entries, "Anna", 0)
  joy = Map.get(entries, "Joy", 0)

  joy = joy + anna

  {joy, %{"Joy" => joy}, ["Anna"]}
end)

@spec get_multi(GenServer.server(), [key()]) :: %{required(key()) => value()}

Gets multiple entries corresponding by the given keys all at once, atomically.

The keys to get are passed as a list. The result is a map of key/value entries corresponding to the given keys. Keys that are not present in the database won't be in the result map.

example

Example

CubDB.put_multi(db, a: 1, b: 2, c: nil)

CubDB.get_multi(db, [:a, :b, :c, :x])
# => %{a: 1, b: 2, c: nil}

@spec halt_compaction(GenServer.server()) :: :ok | {:error, :no_compaction_running}

Stops a running compaction.

If a compaction operation is running, it is halted, and the function returns :ok. Otherwise it returns {:error, :no_compaction_running}. If a new compaction is started (manually or automatically), it will start from scratch, the halted compaction is completely discarded.

This function can be useful if one wants to make sure that no compaction operation is running in a certain moment, for example to perform some write-intensive workload without incurring in additional load. In this case one can pause auto compaction, and call halt_compaction/1 to stop any running compaction.

@spec has_key?(GenServer.server(), key()) :: boolean()

Returns whether an entry with the given key exists in the database.

@spec put(GenServer.server(), key(), value()) :: :ok

Writes an entry in the database, associating key to value.

If key was already present, it is overwritten.

@spec put_and_delete_multi(GenServer.server(), %{required(key()) => value()}, [key()]) ::
  :ok

Writes and deletes multiple entries all at once, atomically.

Entries to put are passed as a map of %{key => value} or a list of {key, value}. Keys to delete are passed as a list of keys.

@spec put_multi(GenServer.server(), %{required(key()) => value()} | [entry()]) :: :ok

Writes multiple entries all at once, atomically.

Entries are passed as a map of %{key => value} or a list of {key, value}.

@spec put_new(GenServer.server(), key(), value()) :: :ok | {:error, :exists}

Writes an entry in the database, associating key to value, only if key is not yet in the database.

If key is already present, it does not change it, and returns {:error, :exists}.

@spec release_snapshot(CubDB.Snapshot.t()) :: :ok

Releases a snapshot when it is not needed anymore, releasing related resources

This allows CubDB to perform cleanup operations after compaction that are otherwise blocked by the snapshot. When creating a snapshot with a timeout, it is not necessary to call release_snapshot/1, as it will be automatically released after the timeout elapses. When getting a snapshot with a timeout of :infinity though, one has to manually call release_snapshot/1 once the snapshot is not needed anymore.

In most cases, using with_snapshot/2 is a better alternative to manually calling snapshot/2 and release_snapshot/1

@spec select(GenServer.server(), [select_option()]) :: Enumerable.t()

Selects a range of entries from the database, returning a lazy stream.

The returned lazy stream can be filtered, mapped, and transformed with standard functions in the Stream and Enum modules. The actual database read is deferred to when the stream is iterated or evaluated.

options

Options

The min_key and max_key specify the range of entries that are selected. By default, the range is inclusive, so all entries that have a key greater or equal than min_key and less or equal then max_key are selected:

# Select all entries where "a" <= key <= "d"
CubDB.select(db, min_key: "b", max_key: "d")

The range boundaries can be excluded by setting min_key_inclusive or max_key_inclusive to false:

# Select all entries where "a" <= key < "d"
CubDB.select(db, min_key: "b", max_key: "d", max_key_inclusive: false)

Any of :min_key and :max_key can be omitted, to leave the range open-ended.

# Select entries where key <= "a"
CubDB.select(db, max_key: "a")

Since nil is a valid key, setting min_key or max_key to nil does NOT leave the range open ended:

# Select entries where nil <= key <= "a"
CubDB.select(db, min_key: nil, max_key: "a")

The reverse option, when set to true, causes the entries to be selected and traversed in reverse order. This is more efficient than selecting them in normal ascending order and then reversing the resulting collection.

Note that, when selecting a key range, specifying min_key and/or max_key is more performant than using functions in Enum or Stream to filter out entries out of range, because min_key and max_key avoid loading unnecessary entries from disk entirely.

examples

Examples

To select all entries with keys between :a and :c as a stream of {key, value} entries we can do:

entries = CubDB.select(db, min_key: :a, max_key: :c)

Since select/2 returns a lazy stream, at this point nothing has been fetched from the database yet. We can turn the stream into a list, performing the actual query:

Enum.to_list(entries)

If we want to get all entries with keys between :a and :c, with :c excluded, we can do:

entries =
  CubDB.select(db,
    min_key: :a,
    max_key: :c,
    max_key_inclusive: false
  )
  |> Enum.to_list()

To select the last 3 entries, we can do:

entries = CubDB.select(db, reverse: true) |> Enum.take(3)

If we want to obtain the sum of the first 10 positive numeric values associated to keys from :a to :f, we can do:

sum =
  CubDB.select(db,
    min_key: :a,
    max_key: :f
  )
  |> Stream.map(fn {_key, value} -> value end) # map values
  |> Stream.filter(fn n -> is_number(n) and n > 0 end) # only positive numbers
  |> Stream.take(10) # take only the first 10 entries in the range
  |> Enum.sum() # sum the selected values

Using functions from the Stream module for mapping and filtering ensures that we do not fetch unnecessary items from the database. In the example above, for example, after fetching the first 10 entries satisfying the filter condition, no further entry is fetched from the database.

@spec set_auto_compact(
  GenServer.server(),
  boolean() | {integer(), integer() | float()}
) ::
  :ok | {:error, String.t()}

Configures whether to perform automatic compaction, and how.

If set to false, no automatic compaction is performed. If set to true, auto-compaction is performed, following a write operation, if at least 100 write operations occurred since the last compaction, and the dirt factor is at least 0.25. These values can be customized by setting the auto_compact option to {min_writes, min_dirt_factor}.

It returns :ok, or {:error, reason} if setting is invalid.

Compaction is performed in the background and does not block other operations, but can create disk contention, so it should not be performed unnecessarily often. When writing a lot into the database, such as when importing data from an external source, it is advisable to turn off auto compaction, and manually run compaction at the end of the import.

@spec set_auto_file_sync(GenServer.server(), boolean()) :: :ok

Configures whether to automatically force file sync upon each write operation.

If set to false, no automatic file sync is performed. That improves write performance, but leaves to the operative system the decision of when to flush disk buffers. This means that there is the possibility that recent writes might not be durable in case of a sudden machine shutdown. In any case, atomicity of multi operations is preserved, and partial writes will not corrupt the database.

If set to true, the file buffer will be forced to flush upon every write operation, ensuring durability even in case of sudden machine shutdowns, but decreasing write performance.

@spec size(GenServer.server()) :: non_neg_integer()

Returns the number of entries present in the database.

@spec snapshot(GenServer.server(), timeout()) :: CubDB.Snapshot.t()

Returns a snapshot of the database in its current state.

Note: it is usually better to use with_snapshot/2 instead of snapshot/2, as the former automatically manages the snapshot life cycle, even in case of crashes.

A snapshot is an immutable, read-only representation of the database at a specific point in time. Getting a snapshot is basically zero-cost: nothing needs to be copied or written, apart from some small in-memory bookkeeping.

The only cost of a snapshot is that it delays cleanup of old files after compaction for as long as it is in use. For this reason, a snapshot has a timeout, configurable as the optional second argument of snapshot/2 (defaulting to 5000, or 5 seconds, if not specified). After such timeout elapses, the snapshot cannot be used anymore, and any pending cleanup is performed.

It is possible to pass :infinity as the timeout, but then one must manually call release_snapshot/1 to release the snapshot after use.

Using with_snapshot/1 is often a better alternative to snapshot/2, as it does not require to choose an arbitrary timeout, and automatically ensures that the the snapshot is released after use, even in case of a crash.

After obtaining a snapshot, it is possible to read from it using the functions in CubDB.Snapshot, which work the same way as the functions in CubDB with the same name, such as CubDB.Snapshot.get/3, CubDB.Snapshot.get_multi/2, CubDB.Snapshot.fetch/2, CubDB.Snapshot.has_key?/2, CubDB.Snapshot.select/2.

It is not possible to write on a snapshot.

example

Example

CubDB.put(db, :a, 123)
snap = CubDB.snapshot(db)

CubDB.put(db, :a, 0)

# Getting a value from the snapshot returns the value of the entry at the
# time the snapshot was obtained, even if the entry has changed in the
# meanwhile
CubDB.Snapshot.get(snap, :a)
# => 123

# Getting the same value from the database returns the latest value
CubDB.get(db, :a)
# => 0

@spec start(String.t() | [option() | {:data_dir, String.t()} | GenServer.option()]) ::
  GenServer.on_start()

Starts the CubDB database without a link.

See start_link/2 for more information about options.

@spec start_link(
  String.t()
  | [option() | {:data_dir, String.t()} | GenServer.option()]
) ::
  GenServer.on_start()

Starts the CubDB database process linked to the current process.

The argument is a keyword list of options:

• data_dir: the directory path where the database files will be stored. This option is required. If the directory does not exist, it will be created. Only one CubDB instance can run per directory, so if you run several databases, they should each use their own separate data directory.

• auto_compact: whether to perform compaction automatically. It defaults to true. See set_auto_compact/2 for the possible values

• auto_file_sync: whether to force flush the disk buffer on each write. It defaults to true. If set to false, write performance is faster, but durability of writes is not strictly guaranteed. See set_auto_file_sync/2 for details.

GenServer options like name and timeout can also be given, and are forwarded to GenServer.start_link/3 as the third argument.

If only the data_dir is specified, it is possible to pass it as a single string argument.

examples

Examples

# Passing only the data dir
{:ok, db} = CubDB.start_link("some/data/dir")

# Passing data dir and other options
{:ok, db} = CubDB.start_link(data_dir: "some/data/dir", auto_compact: true, name: :db)

@spec stop(GenServer.server(), term(), timeout()) :: :ok

Synchronously stops the CubDB database.

See GenServer.stop/3 for details.

@spec transaction(
  GenServer.server(),
  (CubDB.Tx.t() -> {:commit, CubDB.Tx.t(), result} | {:cancel, result})
) :: result
when result: any()

Starts a write transaction, passes it to the given function, and commits or cancels it depending on the return value.

The transaction blocks other writers until the function returns, but does not block concurrent readers. When the need is to only read inside a transaction, and not perform any write, using a snapshot is a better choice, as it does not block writers (see with_snapshot/2).

The module CubDB.Tx contains functions to perform read and write operations within the transaction. The function fun is called with the transaction as argument, and should return {:commit, tx, results} to commit the transaction tx and return result, or {:cancel, result} to cancel the transaction and return result.

If an exception is raised, or a value thrown, or the process exits while inside of a transaction, the transaction is cancelled.

Only use CubDB.Tx functions to write when inside a transaction (like CubDB.Tx.put/3 or CubDB.Tx.delete/2). Using functions in the CubDB module to perform a write when inside a transaction (like CubDB.put/3 or CubDB.delete/2) raises an exception. Note that write functions in CubDB.Tx have a functional API: they return a modified transaction rather than mutating it in place.

The transaction value passed to fun should not be used outside of the function.

example

Example:

Suppose the keys :a and :b map to balances, and we want to transfer 5 from :a to :b, if :a has enough balance:

CubDB.transaction(db, fn tx ->
  a = CubDB.Tx.get(tx, :a)
  b = CubDB.Tx.get(tx, :b)

  if a >= 5 do
    tx = CubDB.Tx.put(tx, :a, a - 5)
    tx = CubDB.Tx.put(tx, :b, b + 5)
    {:commit, tx, :ok}
  else
    {:cancel, :not_enough_balance}
  end
end)

The read functions in CubDB.Tx read the in-transaction state, as opposed to the live database state, so they see writes performed inside the transaction even before they are committed:

# Assuming we start from an empty database
CubDB.transaction(db, fn tx ->
  tx = CubDB.Tx.put(tx, :a, 123)

  # CubDB.Tx.get sees the in-transaction value
  CubDB.Tx.get(tx, :a)
  # => 123

  # CubDB.get instead does not see the uncommitted write
  CubDB.get(db, :a)
  # => nil

  {:commit, tx, nil}
end)

# After the transaction is committed, CubDB.get sees the write
CubDB.get(db, :a)
# => 123

@spec update(GenServer.server(), key(), value(), (value() -> value())) :: :ok

Updates the entry corresponding to key using the given function.

If key is present in the database, fun is invoked with the corresponding value, and the result is set as the new value of key. If key is not found, initial is inserted as the value of key.

@spec with_snapshot(GenServer.server(), (CubDB.Snapshot.t() -> result)) :: result
when result: any()

Calls fun passing a snapshot, and automatically releases the snapshot when the function returns

It returns the value returned by the function fun.

A snapshot is an immutable, read-only representation of the database at a specific point in time, isolated from writes. It is basically zero-cost: nothing needs to be copied or written, apart from some small in-memory bookkeeping.

Calling with_snapshot/2 is equivalent to obtaining a snapshot with snapshot/2 using a timeout of :infinity, calling fun, then manually releasing the snapshot with release_snapshot/1, but with_snapshot/2 automatically manages the snapshot life cycle, also in case an exception is raised, a value is thrown, or the process exists. This makes with_snapshot/2 usually a better choice than snapshot/2.

After obtaining a snapshot, it is possible to read from it using the functions in CubDB.Snapshot, which work the same way as the functions in CubDB with the same name, such as CubDB.Snapshot.get/3, CubDB.Snapshot.get_multi/2, CubDB.Snapshot.fetch/2, CubDB.Snapshot.has_key?/2, CubDB.Snapshot.select/2.

It is not possible to write on a snapshot.

example

Example

Assume that we have two entries in the database, and the key of the second entry depends on the value of the first (so the value of the first entry "points" to the other entry). In this case, we want to get both entries from the same snapshot, to avoid inconsistencies due to concurrent writes. Here's how that can be done with with_snapshot/2:

{x, y} = CubDB.with_snapshot(db, fn snap ->
  x = CubDB.Snapshot.get(snap, :x)
  y = CubDB.Snapshot.get(snap, x)
  {x, y}
end)