Module mnesia_to_khepri

Tools to migrate from Mnesia to Khepri.

Description

Tools to migrate from Mnesia to Khepri.

The migration from Mnesia to Khepri implemented in this module is divided in two distinct parts:
  1. the cluster membership
  2. the tables content

Both parts can be used independently.

Cluster membership synchronization

For the first part, sync_cluster_membership/0 and sync_cluster_membership/1 ensure the default/specified Khepri store has the same cluster members as Mnesia.

All "instances" of the Khepri store on unclustered nodes will be reset except one. The selected surviving Khepri store is determined using several heuristics which are explained in sync_cluster_membership/1.

Tables copy

For the second part, copy_tables/3, copy_tables/4, copy_all_tables/2 and copy_all_tables/3 take care of copying records to the designated Khepri store.

The functions take a converter module which is responsible for actually writing each record to wherever it wants in the Khepri store. This allows the caller to filter and convert the records. The converter module interface is defined by the mnesia_to_khepri_converter behavior. There is an example converter module called mnesia_to_khepri_example_converter provided.

The copy works while Mnesia tables are still being used and updated. To allow that, the copy follows several stages which are explained in copy_tables/4.

The functions is_migration_finished/1, is_migration_finished/2, wait_for_migration/2 and wait_for_migration/3 can be used to follow an on-going copy of the tables.

Finally to help with the use of the Mnesia tables and Khepri store concurrently with a tables copy, you can use handle_fallback/3 and handle_fallback/4.

Data Types

converter_mod()

converter_mod() = module() | {module(), any()}

A converter module, possibly with a private term to initliaze it.

A converter module is a module implementing the mnesia_to_khepri_converter behavior. The private term is passed as is to its {link mnesia_to_khepri_converter:init_copy_to_khepri/3} callback.

migration_id()

migration_id() = binary()

MigrationId of a migration.

This is used to semantically identify a migration that covers a set of Mnesia tables and an associated converter module.

A migration is idempotent, based on this identifier. In other words, a migration identifier by this identifier can happen only once, and there can't be concurrent migration processes with that same identifier.

mnesia_table()

mnesia_table() = atom()

The name of a Mnesia table.

This is the same type as mnesia:table() which is not exported unfortunately.

Function Index

sync_cluster_membership/0Ensures the default Khepri store has the same members as the Mnesia cluster.
sync_cluster_membership/1Ensures the Khepri store named StoreId has the same members as the Mnesia cluster.
copy_tables/3Copies records from Mnesia tables Tables to the default Khepri store.
copy_tables/4Copies records from Mnesia tables Tables to the Khepri store named StoreId.
copy_all_tables/2Copies records from all Mnesia tables to the default Khepri store.
copy_all_tables/3Copies records from all Mnesia tables to the Khepri store named StoreId.
is_migration_finished/1Returns the migration status of the specified migration identifier.
is_migration_finished/2Returns the migration status of the specified migration identifier.
wait_for_migration/2Waits for migration MigrationId to be finish.
wait_for_migration/3Waits for migration MigrationId to be finish.
cleanup_after_table_copy/1Performs any cleanups after a migration has finished.
cleanup_after_table_copy/2Performs any cleanups after a migration has finished.
rollback_table_copy/1Rolls back a migration.
rollback_table_copy/2Rolls back a migration.
handle_fallback/3Runs MnesiaFun or evaluates KhepriFunOrRet depending on the status of the migration.
handle_fallback/4Runs MnesiaFun or evaluates KhepriFunOrRet depending on the status of the migration.

Function Details

sync_cluster_membership/0

sync_cluster_membership() -> ok

Ensures the default Khepri store has the same members as the Mnesia cluster.

See also: sync_cluster_membership/1.

sync_cluster_membership/1

sync_cluster_membership(StoreId) -> ok

Ensures the Khepri store named StoreId has the same members as the Mnesia cluster.

The synchronization is split into the following steps:
  1. Mnesia is queried to list all its members. We ensure that all Mnesia members are running at this point: if some members are down, there is little chance that Khepri will work on those anyway.
  2. All Mnesia members are queried to learn about the membership status of all "instances" of the Khepri store named StoreId.
  3. Among all instances of the Khepri store, one instance is selected to be the one that will be joined by other nodes. The selection process is described below.
  4. Other nodes are reset, meaning their data is dropped, and they are added to the selected Khepri store.

The synchronization process has to select a store instance that will grow and other store instances that will be reset. The criterias below are evaluated to sort the list of store instances, then the first instance in that list is selected as the winning instance.

Criterias are evaluated in order. For a given criteria, if both sides are equal, the next criteria is evaluated until one criteria gives an instance "greater" than another.

Here is the ordered list of criterias:
  1. A Khepri store instance with more members gets precedence.
  2. Then, a Khepri store instance with more tree nodes (more records) gets precedence.
  3. Then, a Khepri store instance where a member's Erlang node runs for the longest time gets precedence.
  4. Then, a Khepri store instance where its members list sorts before another members list gets precedence.
Here are some examples:

copy_tables/3

copy_tables(MigrationId, Tables, Mod) -> Ret

Copies records from Mnesia tables Tables to the default Khepri store.

See also: copy_tables/3.

copy_tables/4

copy_tables(StoreId, MigrationId, Tables, Mod) -> Ret

Copies records from Mnesia tables Tables to the Khepri store named StoreId.

The converter module Mod is responsible for storing each Mnesia record in the Khepri store. How it is called is described below. mnesia_to_khepri_example_converter can be used as the default converter module or as an example to write a new one.

The copy is split into the following steps:
  1. The PID of the Erlang process working on the copy is stored in the Khepri store. This serves as an indicator that tables are being copied and this prevents concurrent copy from happening.
  2. The converter module state is initialized using its Mod:init_copy_to_khepri/3 function, or Mod:init_copy_to_khepri/4 if ModArgs is set.
  3. The copy process subscribes to changes from Mnesia.
  4. The copy process copies records from Mnesia using its Backup & Restore API. The process uses Mod:copy_to_khepri/3 for each Mnesia record.
  5. The copy process marks all involved Mnesia tables as read-only.
  6. The copy process consumes all Mnesia events it received during the initial copy and calls Mod:copy_to_khepri/3 and Mod:delete_from_khepri/3 to update the Khepri store accordingly.
  7. The copy process calls Mod:finish_copy_to_khepri/1 to let the converter module do any cleanups.
  8. The "migration in progress" marker is removed from the Khepri store.
Copied Mnesia tables continue to be available during the process, except after they are marked as read-only. See handle_fallback/2 and handle_fallback/3 for helpers to use Mnesia while records are being copied.

copy_all_tables/2

copy_all_tables(MigrationId, Mod) -> Ret

Copies records from all Mnesia tables to the default Khepri store.

See also: copy_all_tables/3.

copy_all_tables/3

copy_all_tables(StoreId, MigrationId, Mod) -> Ret

Copies records from all Mnesia tables to the Khepri store named StoreId.

See also: copy_tables/4.

is_migration_finished/1

is_migration_finished(MigrationId) -> Migrated

Returns the migration status of the specified migration identifier.

The default Khepri store is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.

See also: is_migration_finished/2.

is_migration_finished/2

is_migration_finished(StoreId, MigrationId) -> Migrated

returns: true if the migration is finished, {in_flight, Pid} if the migration is in progress and handled by the Pid process, false if the migration has not started or undefined if the query of the Khepri store where the status is recorded failed.

Returns the migration status of the specified migration identifier.

The Khepri store named StoreId is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.

wait_for_migration/2

wait_for_migration(MigrationId, Timeout) -> Ret

Waits for migration MigrationId to be finish.

The default Khepri store is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.

See also: wait_for_migration/3.

wait_for_migration/3

wait_for_migration(StoreId, MigrationId, Timeout) -> Ret

Waits for migration MigrationId to be finish.

If the migration has not started, it returns false immediately.

If the migration is finished, it returns true immediately.

If the migration is in progress or the status is undefined (see is_migration_finished/3), it waits until the status is known to be "finished" or "not started" or until Timeout milliseconds.

The Khepri store named StoreId is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.

cleanup_after_table_copy/1

cleanup_after_table_copy(MigrationId) -> Ret

Performs any cleanups after a migration has finished.

See also: cleanup_after_table_copy/2.

cleanup_after_table_copy/2

cleanup_after_table_copy(StoreId, MigrationId) -> Ret

Performs any cleanups after a migration has finished.

Currently this includes the deletion of the copied Mnesia tables.

This is a separate step from copy_tables/4 because the caller might want to record some post-migration states before comitting to delete the Mnesia tables.

rollback_table_copy/1

rollback_table_copy(MigrationId) -> Ret

Rolls back a migration.

See also: rollback_table_copy/2.

rollback_table_copy/2

rollback_table_copy(StoreId, MigrationId) -> Ret

Rolls back a migration.

This function puts Mnesia tables back to read-write mode and deletes the "migration in progress" marker.

Rollback is not possible once cleanup_after_table_copy/2 is used.

handle_fallback/3

handle_fallback(MigrationId, MnesiaFun, KhepriFunOrRet) -> Ret

Runs MnesiaFun or evaluates KhepriFunOrRet depending on the status of the migration.

The default Khepri store is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.

See also: handle_fallback/4.

handle_fallback/4

handle_fallback(StoreId, MigrationId, MnesiaFun, KhepriFunOrRet) -> Ret

Runs MnesiaFun or evaluates KhepriFunOrRet depending on the status of the migration.

If the migration is finished it executes KhepriFunOrRet if it is a function with an arity of 0 or returns the term directly otherwise.

If the migration is not finished, the function tries to execute MnesiaFun. It should fail by returning or raising {aborted, {no_exists, Table}}. When this happens, the function waits for the migration to finish using wait_for_table_migration/3, then it starts again.

The Khepri store named StoreId is queried to get the migration status. It must correspond to the Khepri store passed to copy_tables/3 and similar functions.


Generated by EDoc