Modify the database through static migrations.

Each migrations typically have an up and down function or just a change function. If using separate up and down functions, these will allow for the database to migrated forward or rolled back. If using the change function then the migrations for going forward are defined and automatically reversed in the case of a rollback.

Migrations that have been executed in a database for ArangoXEcto are stored in a system collection called _migrations (this is a hidden system collection). This allows ArangoXEcto to track what has already been run so that it knows what migrations can be run or rolled back. You can configure the name of this table with the :migration_source configuration option and the name of the repository that manages migrations can be configured with :migration_repo.

Two (or more) servers, one DB, oh no!!

You might be wondering what happens when there are two or more applications trying to run migrations on the database at once. If two or more instances of the database try to write to the database at the same time this could be a huge problem, causing unpredictable results. A lot of databases use locking as a mechanism to solve this. I.e. one connection will lock writing to the database so others cannot write. ArangoDB doesn't do this, so how can we solve this?

Luckily we don't have to worry about this as ArangoDB handles this for us through dynamic locks. Essentially if two connections try to write on the same part of the database then one of them will just fail as the first has acquired an exclusive lock.

Migration creation

Migration files are stored in the "priv/MY_REPO/migrations" (where MY_REPO is a lowercase underscored name of your repository. For example, if you repo module was MyApp.MyRepo then the path would be "priv/my_repo/migrations" or if your repo module was MyApp.ThisIsTheCoolestRepo then your migrations would be in "priv/this_is_the_coolest_repo/migrations". The directory can be overridden when running the mix tasks or in the configuration.

Each migration filename will have a timestamp number followed by a unique name seperated by an underscore. For example if you wanted to add a users collection could put the migration in a file at "priv/repo/migrations/20240928200000_add_users_collection.exs, with contents:

defmodule Repo.Migrations.AddUsersCollection do
  use ArangoXEcto.Migration

  def up do
    create collection(:users) do
      add :first_name, :string, comment: "first_name column"
      add :last_name, :string
      add :gender, :integer
      add :age, :integer
      add :location, :map

      timestamps()
    end
  end

  def down do
    drop collection("users")
  end
end

The up/0 function is responsible for migrating the database forward, while the down/0 function is responsible for rolling back this migration. The down/0 function must always reverse the up/0 action. Inside both of these functions we use various API methods defined in this module to perform actions on the database. We can manage collections, indexes, views, analyzers and run custom AQL commands.

Now that our migration is setup, all that is left is to execute it. While migrations can be run using code (and there are valid use cases for this, e.g. generating releases for production) we generally will use mix tasks. To do this run the following from the root directory of your project:

$ mix arango.migrate

If we wanted to reverse the migration we could run:

$ mix arango.rollback -n 1

We must always specify how many migrations to rollback. Whereas mix arango.migrate will always run all pending migrations

There is one other mix task that is used regularly and it saves us a lot of time. That is the mix arango.gen.migration task. This task will generate a blank migration with boilerplate contents and use the proper timestamps of when it was generated.

$ mix arango.gen.migration add_users_collection

You can now generate and run migrations successfully. You may be pretty proud of yourself, and you should be, but there is some more parts to it. You can learn all about migrations in ArangoXEcto in this module documentation.

ArangoDB Schemas

ArangoDB can take a schema, but it is not strictly required, hence no fields need to be provided. The only required thing is the collection name. More info below on how to define fields. For example, the example provided in the previous section can be provided without defining the fields.

defmodule Repo.Migrations.AddUsersCollection do
  use ArangoXEcto.Migration

  def up do
    create collection(:users)
  end

  def down do
    drop collection("users")
  end
end

This will work perfectly fine, however there is one caveat, documents in the collection will not be validated. This means that the migration will only create the collection and not define a JSONSchema for the collection. Schemas a generated using the ArangoXEcto.Migration.JsonSchema module, however all documentation on how to use it can be found in this module.

Mix tasks

As seen in the example above, ArangoXEcto has a few built in mix tasks that can be used to help with the development workflow. These tasks are similar to the ones found in ecto_sql but are adapted for the ArangoXEcto context. To allow for using multiple database adapters together they follow a similar but slightly different pattern, as seen below.

• mix arango.gen.migration - generates a migration that the user can fill in with particular commands
• mix arango.migrate - migrates a repository
• mix arango.rollback - rolls back a particular migration

For additional help on how to use these tasks you can run mix help COMMAND where COMMAND is the command you need assistance with.

As mentioned previously, you can also run migrations using code using this module. If you think you're pretty cool and know what you're doing you can also use the lower level APIs in ArangoXEcto.Migrator.

Change (combining up and down)

I know what you're thinking, "Ugh, do I really have to write an up/0 and down/0 function for every single migration? Surely you could have just written some code the generate the down/0 version for me". Well yes, yes I did. That's what change/0 is for. You just write your regular up/0 action and it will do some fancy footwork and figure out what the down/0 version would be. For example, this would operate the same as seen above:

defmodule Repo.Migrations.AddUsersCollection do
  use ArangoXEcto.Migration

  def change do
    create collection(:users) do
      add :first_name, :string, comment: "first_name column"
      add :last_name, :string
      add :gender, :integer
      add :age, :integer
      add :location, :map

      timestamps()
    end
  end
end

Like a good salesman I only gave you half the story, there is one caveat when using this. Not all commands are reversible and will raise an Ecto.MigrationError.

A great example of a command that cannot be reversed is execute/1 but can easily be made reversible by calling execute/2 instead. The first argument will be run on up/0 and the second will be run on down/0.

If you implement up/0 and down/0 they will take precedence over change/0 which won't be invoked.

Field Types

The field types provided must match one of the types available in ArangoXEcto.Migration.JsonSchema. Any other option will raise an error. There is one caveat to this, the sub_command option is used through add_embed and add_embed_many (essentially a shortcut for an array of many embeds).

The options that can be passed to each of the types can be found in the ArangoXEcto.Migration.JsonSchema docs.

Executing and flushing

Most functions won't be executed until the end of the relevant up, change or down callback. However, if you call any Ecto.Repo function they will be executed immediately, unless called inside an anonymous function passed to execute/1.

You may want to ensure that all the previous steps have been executed before continuing. To do this you can call the flush/0 function to execute everything before.

However flush/0 will raise if it would be called from change function when doing a rollback. To avoid that we recommend to use execute/2 with anonymous functions instead. For more information and example usage please take a look at execute/2 function.

Repo configuration

Migrator configuration

The following options are not required but can be used to adjust how your migrations are stored and run.

• :migration_source - The name of the collection to store the migrations that have already been run. Only the version numbers (timestamps) are stored. The default is _migrations. Whatever is specified it will be set as a system collection in ArangoDB. The convention is to prefix the collection name with an underscore (_). You can configure this as follows:

  config :my_app, MyApp.Repo, migration_source: "_custom_migrations"

• :migration_repo - This configures which repository the migrations will be stored in. By default this will be the given repository but it can be overridden. A possible use case of this is if you wanted all migrations to be stored in a particular database in ArangoDB.

  config :my_app, MyApp.Repo, migration_repo: MyApp.MigrationRepo

• :priv - the location where to store important assets, such as migrations. By default this will be "priv/my_reoo" for a repository module of MyApp.MyRepo and migrations would be placed in "priv/my_repo/migrations".

• :start_apps_before_migration - A list of applications to be started before migrations are run. Used by ArangoXEcto.Migrator.with_repo/3 and hence the migration task.

  config :my_app, MyApp.Repo, start_apps_before_migration: [:ssl, :some_custom_logger]

Migrations configuration

The previous section's options were focused on how migrations are run and stored, whereas the following options are focused on adjusting what values are stored. If you change these after your app goes into production it will cause unexpected behaviour.

• :migration_timestamps - Modifies the type and field names of :inserted_at and :updated_at. By default timestamps will be stored as :naive_datetime but this, and the field names can be configured as seen below:

  config :my_app, MyApp.Repo, migration_timestamps: [
    type: :utc_datetime,
    inserted_at: :created_at,
    updated_at: :changed_at
  ]

• :migration_default_prefix - By default no prefix is used, but you might want to. You can configure it using this option. This will be overriden by a prefix being passed at the command level.

  config :my_app, MyApp.Repo, migration_default_prefix: "my_migrations_prefix"

Comments

The schema stored in ArangoDB supports storing comments on the fields. You can specify this by passing the :comment option when using the add/3, add_embed/3, add_embed_many/3 and the modify equivalents.

  def up do
    create collection(:users) do
      add :name, :string, comment: "full name column"
      add_embed :facts, comment: "my facts" do
        add :statement, :string, comment: "represents the fact statement"
        add :falsehood, :boolean, comment: "is the fact false?"
      end

      timestamps()
    end
  end

Prefixes

ArangoXEcto fully supports prefixes, which includes in migrations. Unlike in a database provider like PostgreSQL, ArangoDB doesn't have the notion of prefixes, hence prefixes actually create a separate database to create the separation.

  def up do
    create collection(:users, prefix: "base_app") do
      add :first_name, :string, comment: "first_name column"
      add :last_name, :string
      add :gender, :integer
      add :age, :integer
      add :location, :map

      timestamps()
    end

    create index(:users, [:age], prefix: "base_app")
  end

Notice here how the same prefix must be specified on both the collection creation and the index creation. If you don't do this then they simply won't be created in the same database and will likely result in an error. You can specify a default prefix, as mentioned above, using the :migration_default_prefix in your configuration.

Transaction Callbacks

Migrations are run in transactions. You may need to perform some actions after beginning a transaction or before committing the migration. You can do this with the after_begin/0 and before_commit/0 callbacks to your migration.

Sometimes you may want to use these callbacks for every migration and doing so can be quite repetitive. You can solve this by implementing your own migration module that extends the ArangoXEcto.Migration module:

defmodule MyApp.Migration do
  defmacro __using__(_) do
    quote do
      use ArangoXEcto.Migration

      def after_begin() do
        repo().query! "SOME ARANGO QUERY"
      end
    end
  end
end

Then in your migrations you can replace use ArangoXEcto.Migration with use MyApp.Migration.

Example

defmodule MyProject.Repo.Migrations.CreateUsers do
  use ArangoXEcto.Migration

  def change do
    create analyzer(:norm_en, :norm, [:frequency, :position], %{
       locale: "en",
       accent: false,
       case: :lower
     })

    create collection(:users) do
      add :first_name, :string, comment: "first_name column"
      add :last_name, :string
      add :gender, :integer
      add :age, :integer
      add :location, :map

      timestamps()
    end

    create index(:users, [:age])
    create unique_index(:users, [:location])

    create view(:user_search, commitIntervalMsec: 1, consolidationIntervalMsec: 1) do
      add_sort(:created_at, :desc)
      add_sort(:first_name)

      add_store([:first_name, :last_name], :none)

      add_link("users", %Link{
        includeAllFields: true,
        fields: %{
          last_name: %Link{
            analyzers: [:identity, :norm_en]
          }
        }
      })
    end
  end
end