Ecto integration with MongoDB.

This document will present a general overview of using MongoDB with Ecto, including common pitfalls and extra functionalities.

Check the Ecto documentation for an introduction or examples/simple for a sample application using Ecto and MongoDB.

Repositories

The first step to use MongoDB with Ecto is to define a repository with Mongo.Ecto as an adapter. First define a module:

defmodule Repo do
  use Ecto.Repo, otp_app: :my_app
end

Then configure it your application environment, usually in your config/config.exs:

config :my_app, Repo,
  adapter: Mongo.Ecto,
  database: "ecto_simple",
  username: "mongodb",
  password: "mongodb",
  hostname: "localhost"

Each repository in Ecto defines a start_link/0 function that needs to be invoked before using the repository. This function is generally from your supervision tree:

def start(_type, _args) do
  import Supervisor.Spec

  children = [
    worker(Repo, [])
  ]

  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Models

With the repository defined, we can define our models:

defmodule Weather do
  use Ecto.Model

  # see the note below for explanation of that line
  @primary_key {:id, :binary_id, autogenerate: true}

  # weather is the MongoDB collection name
  schema "weather" do
    field :city,    :string
    field :temp_lo, :integer
    field :temp_hi, :integer
    field :prcp,    :float, default: 0.0
  end
end

Ecto defaults to using :id type for primary keys, that is translated to :integer for SQL databases, and is not handled by MongoDB. You need to specify the primary key to use the :binary_id type, that the adapter will translate to ObjectID. Remember to place this declaration before the schema call.

The name of the primary key is just a convenience, as MongoDB forces us to use _id. Every other name will be recursively changed to _id in all calls to the adapter. We propose to use id or _id as your primary key name to limit eventual confusion, but you are free to use whatever you like. Using the autogenerate: true option will tell the adapter to take care of generating new ObjectIDs. Otherwise you need to do this yourself.

Since setting @primary_key for every model can be too repetitive, we recommend you to define your own module that properly configures it:

defmodule MyApp.Model do
  defmacro __using__(_) do
    quote do
      use Ecto.Model
      @primary_key {:id, :binary_id, autogenerate: true}
      @foreign_key_type :binary_id # For associations
    end
  end
end

Now, instead of use Ecto.Model, you can use MyApp.Model in your modules. All Ecto types, except :decimal, are supported by Mongo.Ecto.

By defining a schema, Ecto automatically defines a struct with the schema fields:

iex> weather = %Weather{temp_lo: 30}
iex> weather.temp_lo
30

The schema also allows the model to interact with a repository:

iex> weather = %Weather{temp_lo: 0, temp_hi: 23}
iex> Repo.insert!(weather)
%Weather{...}

After persisting weather to the database, it will return a new copy of %Weather{} with the primary key (the id) set. We can use this value to read a struct back from the repository:

# Get the struct back
iex> weather = Repo.get Weather, "507f191e810c19729de860ea"
%Weather{id: "507f191e810c19729de860ea", ...}

# Update it
iex> weather = %{weather | temp_lo: 10}
iex> Repo.update!(weather)
%Weather{...}

# Delete it
iex> Repo.delete!(weather)
%Weather{...}

Queries

Mongo.Ecto also supports writing queries in Elixir to interact with your MongoDB. Let's see an example:

import Ecto.Query, only: [from: 2]

query = from w in Weather,
      where: w.prcp > 0 or is_nil(w.prcp),
     select: w

# Returns %Weather{} structs matching the query
Repo.all(query)

Queries are defined and extended with the from macro. The supported keywords in MongoDB are:

• :where
• :order_by
• :offset
• :limit
• :select
• :preload

When writing a query, you are inside Ecto's query syntax. In order to access params values or invoke functions, you need to use the ^ operator, which is overloaded by Ecto:

def min_prcp(min) do
  from w in Weather, where: w.prcp > ^min or is_nil(w.prcp)
end

Besides Repo.all/1, which returns all entries, repositories also provide Repo.one/1, which returns one entry or nil, and Repo.one!/1 which returns one entry or raises.

There is also support for count function in queries that uses MongoDB's count command. Please note that unlike in SQL databases you can only select a count - there is no support for querying using a count, there is also no support for counting documents and selecting them at the same time.

Please note that not all Ecto queries are valid MongoDB queries. The adapter will raise Ecto.QueryError if it encounters one, and will try to be as specific as possible as to what exactly is causing the problem.

For things that are not possible to express with Elixir's syntax in queries, you can use keyword fragments:

from p in Post, where: fragment(key:["$exists": true]), select: p

To ease of using in more advanced queries, there is Mongo.Ecto.Helpers module you could import into modules dealing with queries. Please see the documentation of the Mongo.Ecto.Helpers module for more information and supported options.

Options for reader functions (Repo.all/2, Repo.one/2, etc)

Such functions also accept options when invoked which allow you to use parameters specific to MongoDB find function:

• :slave_ok - the read operation may run on secondary replica set member
• :partial - partial data from a query against a sharded cluster in which some shards do not respond will be returned in stead of raising error

Upserts

MongoDB's upsert featureset is not as rich as that found in other databases, and therefore not all features provided for by Ecto are supported.

Some upsert operations can be achieved with multiple database calls and are therefore unsafe. A warning is printed to the console in these cases. Since these operations involve multiple DB calls they cannot be considered safe and cannot be recommended. They're intended mostly to make the transition for Ecto developers coming from another database as clean as possible.

Other operations are completely impossible without jumping through many database calls and at the time of writing it wasn't deemed worth it to take this much further. In these cases an error is raised and it's up to you to pick another strategy. Of course PRs are always welcome!

https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2-upserts

Commands

MongoDB has many administrative commands you can use to manage your database. We support them through the Mongo.Ecto.command/2 function.

Mongo.Ecto.command(MyRepo, createUser: "ecto", ...)

We also support one higher level command - Mongo.Ecto.truncate/1 that is used to clear the database, i.e. during testing.

Mongo.Ecto.truncate(MyRepo)

You can use it in your setup call for cleaning the database before every test. You can define your own module to use instead of ExUnit.Case, so you don't have to define this each time.

defmodule MyApp.Case do
  use ExUnit.CaseTemplate

  setup do
    Mongo.Ecto.truncate(MyRepo)
    :ok
  end
end

Please see documentation for those functions for more information.

Associations

Ecto supports defining associations on schemas:

defmodule Post do
  use Ecto.Model

  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id

  schema "posts" do
    has_many :comments, Comment
  end
end

Keep in mind that Ecto associations are stored in different Mongo collections and multiple queries may be required for retrieving them.

While Mongo.Ecto supports almost all association features in Ecto, keep in mind that MongoDB does not support joins as used in SQL - it's not possible to query your associations together with the main model.

Some more elaborate association schemas may force Ecto to use joins in some queries, that are not supported by MongoDB as well. One such call is Ecto.Model.assoc/2 function with a has_many :through association.

You can find more information about defining associations and each respective association module in Ecto.Schema docs.

Embedded models

Ecto supports defining relations using embedding models directly inside the parent model, and that fits MongoDB's design perfectly.

defmodule Post do
  #...

  schema "posts" do
    embeds_many :comments, Comment
  end
end

defmodule Comment do
  embedded_schema do
    field :body, :string
  end
end

You can find more information about defining embedded models in the Ecto.Schema docs.

Indexes and Migrations

Although schema migrations make no sense for databases such as MongoDB there is one field where they can be very beneficial - indexes. Because of this Mongodb.Ecto supports Ecto's database migrations. You can generate a migration with:

$ mix ecto.gen.migration create_posts

This will create a new file inside priv/repo/migrations with the up and down functions. Check Ecto.Migration for more information.

Because MongoDB does not support (or need) database schemas majority of the functionality provided by Ecto.Migration is not useful when working with MongoDB. As we've already noted the most useful part is indexing, but there are others - creating capped collections, executing administrative commands, or migrating data, e.g.:

defmodule SampleMigration do
  use Ecto.Migration

  def up do
    create table(:my_table, options: [capped: true, size: 1024])
    create index(:my_table, [:value])
    create unique_index(:my_table, [:unique_value])
    execute touch: "my_table", data: true, index: true
  end

  def down do
    # ...
  end
end

MongoDB adapter does not support create_if_not_exists or drop_if_exists migration functions.

MongoDB adapter features

The adapter uses mongodb for communicating with the database and a pooling library such as poolboy for managing connections.

The adapter has support for:

• documents with ObjectID as their primary key
• insert, find, update, remove and count mongo functions
• management commands with command/2
• embedded documents either with :map type, or embedded models
• partial updates using change_map/2 and change_array/2 from the Mongo.Ecto.Helpers module
• queries using javascript expression and regexes using respectively javascript/2 and regex/2 functions from Mongo.Ecto.Helpers module.

MongoDB adapter options

Options passed to the adapter are split into different categories decscribed below. All options should be given via the repository configuration.

Compile time options

Those options should be set in the config file and require recompilation in order to make an effect.

• :adapter - The adapter name, in this case, Mongo.Ecto
• :pool - The connection pool module, defaults to Mongo.Pool.Poolboy
• :log_level - The level to use when logging queries (default: :debug)

Connection options

• :hostname - Server hostname (default: localhost)
• :port - Server port (default: 27017)
• :username - Username
• :password - User password
• :mongo_url - A MongoDB URL
• :connect_timeout - The timeout for establishing new connections (default: 5000)
• :w - MongoDB's write convern (default: 1). If set to 0, some of the Ecto's functions may not work properely
• :j, :fsync, :wtimeout - Other MongoDB's write concern options. Please consult MongoDB's documentation

Pool options

Mongo.Ecto does not use Ecto pools, instead pools provided by the MongoDB driver are used. The default poolboy adapter accepts following options:

• :pool_size - The number of connections to keep in the pool (default: 10)
• :max_overflow - The maximum overflow of connections (default: 0)

For other adapters, please see their documentation.