This module provides some boilerplate code for a better support of structs while using the MongoDB driver:

• automatic load and dump function
• reflection functions
• type specification
• support for embedding one and many structs
• support for after load function
• support for before dump function
• support for id generation
• support for default values
• support for derived values

When using the MongoDB driver only maps and keyword lists are used to represent documents. If you prefer to use structs instead of the maps to give the document a stronger meaning or to emphasize its importance, you have to create a defstruct and fill it from the map manually:

defmodule Label do
  defstruct name: "warning", color: "red"
end

iex> label_map = Mongo.find_one(:mongo, "labels", %{})
%{"name" => "warning", "color" => "red"}
iex> label = %Label{name: label_map["name"], color: label_map["color"]}

We have defined a module Label as defstruct, then we get the first label document the collection labels. The function find_one returns a map. We convert the map manually and get the desired struct.

If we want to save a new structure, we have to do the reverse. We convert the struct into a map:

iex> label = %Label{}
iex> label_map = %{"name" => label.name, "color" => label.color}
iex> {:ok, _} = Mongo.insert_one(:mongo, "labels", label_map)

Alternatively, you can also remove the __struct__ key from label. The MongoDB driver automatically converts the atom keys into strings.

iex>  Map.drop(label, [:__struct__])
%{color: :red, name: "warning"}

If you use nested structures, the work becomes a bit more complex. In this case, you have to use the inner structures convert manually, too.

If you take a closer look at the necessary work, two basic functions can be derived:

• load Conversion of the map into a struct.
• dump Conversion of the struct into a map.

This module provides the necessary macros to automate this boilerplate code. The above example can be rewritten as follows:

defmodule Label do

  use Collection

  document do
    attribute :name, String.t(), default: "warning"
    attribute :color, String.t(), default: :red
  end

end

This results in the following module:

defmodule Label do

  defstruct [name: "warning", color: "red"]

  @type t() :: %Label{String.t(), String.t()}

  def new()...
  def load(map)...
  def dump(%Label{})...
  def __collection__(:attributes)...
  def __collection__(:types)...
  def __collection__(:collection)...
  def __collection__(:id)...

end

You can now create new structs with the default values and use the conversion functions between maps and structs:

iex(1)> x = Label.new()
%Label{color: :red, name: "warning"}
iex(2)> m = Label.dump(x)
%{color: :red, name: "warning"}
iex(3)> Label.load(m, true)
%Label{color: :red, name: "warning"}

The load/2 function distinguishes between keys of type binaries load(map, false) and keys of type atoms load(map, true). The default is load(map, false):

iex(1)> m = %{"color" => :red, "name" => "warning"}
iex(2)> Label.load(m)
%Label{color: :red, name: "warning"}

If you would now expect atoms as keys, the result of the conversion is not correct in this case:

iex(3)> Label.load(m, true)
%Label{color: nil, name: nil}

The background is that MongoDB always returns binaries as keys and structs use atoms as keys.

Dump and load function

Using the collection modules increases the cpu usage because we need to serialize the map into the struct if we load the document from the database. And if we want to save the struct to the database, the dump function is called, that serializes the struct into a map. Both functions take care about the key mapping as well. What do the generated functions look like in detail?

Assuming we have the following collections defined:

defmodule Label do
  use Collection
  document do
    attribute :name, String.t(), default: "warning"
  end
end

defmodule User do
  use Collection
  document do
    attribute :first_name, String.t()
    attribute :last_name, String.t()
    attribute :email, String.t()
    embeds_one :label, Label
  end
end

The collection module creates the load function like this:

Map.put(%{

 first_name: Map.get(user, "first_name"),
 last_name: Map.get(user, "last_name"),
 email: Map.get(user, "email"),
 label: Label.load(Map.get(user, "label"))

}, :struct, User)

The performance depends on the number of attributes and embedded structs. It seems, that using a map first is the fasted way to create a new struct. A few variations were tried out:

new quoted load 2.43 M load manually created using Map.get 2.25 M - 1.08x slower +32.91 ns load manually created using [] 2.12 M - 1.15x slower +60.34 ns original load 0.82 M - 2.96x slower +805.70 ns

The dump function looks like this:

[
  {"first_name", user.first_name},
  {"last_name", user.last_name},
  {"email", user.email},
  {"label", Label.dump(user.label)}
]
|> Enum.filter(fn {_key, value} -> value != nil end)
|> Map.new()

The load and dump function uses only the defined attributes and embedded structs. Unknown attributes are ignored.

Default and derived values

Attributes have two options:

• default: a value or a function, which is called, when a new struct is created
• derived: true to indicate, that is attribute should not be saved to the database

If you call new/0 a new struct is returned filled with the default values. In case of a function the function is called to use the return value as default.

  attribute: created, DateTime.t(), &DateTime.utc_now/0

If you mark an attribute as a derived attribute (derived: true) then the dump function will remove the attributes from the struct automatically for you, so these kind of attributes won't be saved in the database.

  attribute :id, String.t(), derived: true

Key mapping

It is possible to define a different name for the keys. The name of the key is defined by using the :name option.

attribute :very_long_name, String.t(), name: v

The dump and load functions will use the name v instead of very_long_name:

defmodule Label do
  use Mongo.Collection
  document do
    attribute :very_long_attribute, String.t(), name: :v
  end
end

iex > l = %Label{very_long_attribute: "Hello"}
%Label{very_long_attribute: "Hello"}
iex > Label.dump(l)
%{"v" => "Hello"}
iex > dumped_l = Label.dump(l)
%{"v" => "Hello"}
iex > Label.load(dumped_l)
%Label{very_long_attribute: "Hello"}

You can reduce the size of the collection by using short attribute-names or use the :name option to import the documents for a migration.

Collections

In MongoDB, documents are written in collections. We can use the collection/2 macro to create a collection:

  defmodule Card do

    use Collection

    collection "cards" do
      attribute :title, String.t(), default: "new title"
    end

  end

The collection/2 macro creates a collection that is basically similar to a document, where an attribute for the ID is added automatically. Additionally, the attribute @collection is assigned and can be used as a constant in other functions.

In the example above we only suppress a warning of the editor by @collection. The macro creates the following expression: @collection "cards". By default, the following attribute is created for the ID:

{:_id, BSON.ObjectId.t(), &Mongo.object_id/0}

where the default value is created via the function &Mongo.object_id/0 when calling new/0:

  iex> Card.new()
  %Card{_id: #BSON.ObjectId<5ec3d04a306a5f296448a695>, title: "new title"}

Two additional reflection features are also provided:

  iex> Card.__collection__(:id)
  :_id
  iex(3)> Card.__collection__(:collection)
  "cards"

MongoDB example

We define the following collection:

  defmodule Card do

    use Collection

    @collection nil ## keeps the editor happy
    @id nil

    collection "cards" do
      attribute :title, String.t(), default: "new title"
    end

    def insert_one(%Card{} = card) do
      with map <- dump(card),
           {:ok, _} <- Mongo.insert_one(:mongo, @collection, map) do
        :ok
      end
    end

    def find_one(id) do
      :mongo
      |> Mongo.find_one(@collection, %{@id => id})
      |> load()
    end

  end

Then we can call the functions insert_one and find_one. Thereby we always use the defined structs as parameters or get the struct as result:

iex(1)> card = Card.new()
%Card{_id: #BSON.ObjectId<5ec3ed0d306a5f377943c23c>, title: "new title"}
iex(6)> Card.insert_one(card)
:ok
iex(2)> Card.find_one(card._id)
%XCard{_id: #BSON.ObjectId<5ec3ecbf306a5f3779a5edaa>, title: "new title"}

ID generator

In MongoDB it is common to use the attribute _id as id. The value is uses an ObjectId generated by the mongodb driver. This behavior can be specified by the module attribute @id_generator when using collection. The default setting is

  {:_id, BSON.ObjectId.t(), &Mongo.object_id/0}

Now you can overwrite this tuple {name, type, function} as you like:

  @id_generator false # no ID creation
  @id_generator {id, String.t, &IDGenerator.next()/0} # customized name and generator
  @id_generator nil # use default: {:_id, BSON.ObjectId.t(), &Mongo.object_id/0}

Embedded documents

Until now we had only shown simple attributes. It will only be interesting when we embed other structs. With the macros embeds_one/3 and embeds_many/3, structs can be added to the attributes:

Example embeds_one

  defmodule Label do

    use Collection

    document do
      attribute :name, String.t(), default: "warning"
      attribute :color, String.t(), default: :red
    end

  end

  defmodule Card do

    use Collection

    collection "cards" do
      attribute   :title, String.t()
      attribute   :list, BSON.ObjectId.t()
      attribute   :created, DateString.t(), default: &DateTime.utc_now/0
      attribute   :modified, DateString.t(), default: &DateTime.utc_now/0
      embeds_one  :label, Label, default: &Label.new/0
    end

  end

If we now call new/0, we get the following structure:

  iex(1)> Card.new()
  %Card{
    _id: #BSON.ObjectId<5ec3f0f0306a5f3aa5418a24>,
    created: ~U[2020-05-19 14:45:04.141044Z],
    label: %Label{color: :red, name: "warning"},
    list: nil,
    modified: ~U[2020-05-19 14:45:04.141033Z],
    title: nil
  }

after_load/1 and before_dump/1 macros

Sometimes you may want to perform post-processing after loading the data set, for example to create derived attributes. Conversely, before saving, you may want to drop the derived attributes so that they are not saved to the database.

For this reason there are two macros after_load/1 and before_dump/1. You can specify functions that are called after the load/0 or before the dump:

Example embeds_many

  defmodule Board do

    use Collection

    collection "boards" do

      attribute   :id, String.t() ## derived attribute
      attribute   :title, String.t()
      attribute   :created, DateString.t(), default: &DateTime.utc_now/0
      attribute   :modified, DateString.t(), default: &DateTime.utc_now/0
      embeds_many :lists, BoardList

      after_load  &Board.after_load/1
      before_dump &Board.before_dump/1
    end

    def after_load(%Board{_id: id} = board) do
      %Board{board | id: BSON.ObjectId.encode!(id)}
    end

    def before_dump(board) do
      %Board{board | id: nil}
    end

    def new(title) do
      new()
      |> Map.put(:title, title)
      |> Map.put(:lists, [])
      |> after_load()
    end

    def store(board) do
      with map <- dump(board),
          {:ok, _} <- Mongo.insert_one(:mongo, @collection, map) do
        :ok
      end
    end

    def fetch(id) do
      :mongo
      |> Mongo.find_one(@collection, %{@id => id})
      |> load()
    end

  end

In this example the attribute id is derived from the actual ID and stored as a binary. This attribute is often used and therefore we want to save the conversion of the ID. To avoid storing the derived attribute id, the before_dump/1 function is called, which removes the id from the struct:

  iex(1)> board = Board.new("Vega")
  %Board{
    _id: #BSON.ObjectId<5ec3f802306a5f3ee3b71cf2>,
    created: ~U[2020-05-19 15:15:14.374556Z],
    id: "5ec3f802306a5f3ee3b71cf2",
    lists: [],
    modified: ~U[2020-05-19 15:15:14.374549Z],
    title: "Vega"
  }
  iex(2)> Board.store(board)
  :ok
  iex(3)> Board.fetch(board._id)
  %Board{
    _id: #BSON.ObjectId<5ec3f802306a5f3ee3b71cf2>,
    created: ~U[2020-05-19 15:15:14.374Z],
    id: "5ec3f802306a5f3ee3b71cf2",
    lists: [],
    modified: ~U[2020-05-19 15:15:14.374Z],
    title: "Vega"
  }

If we call the document in the Mongo shell, we see that the attribute id was not stored there:

  > db.boards.findOne({"_id" : ObjectId("5ec3f802306a5f3ee3b71cf2")})
  {
    "_id" : ObjectId("5ec3f802306a5f3ee3b71cf2"),
    "created" : ISODate("2020-05-19T15:15:14.374Z"),
    "lists" : [ ],
    "modified" : ISODate("2020-05-19T15:15:14.374Z"),
    "title" : "Vega"
  }

Example timestamps

  defmodule Post do

    use Mongo.Collection

    collection "posts" do
      attribute :title, String.t()
      timestamps()
    end

    def new(title) do
      Map.put(new(), :title, title)
    end

    def store(post) do
      MyRepo.insert_or_update(post)
    end
  end

In this example the macro timestamps is used to create two DateTime attributes, inserted_at and updated_at. This macro is intented to use with the Repo module, as it will be responsible for updating the value of updated_at attribute before execute the action.

  iex(1)> post = Post.new("lorem ipsum dolor sit amet")
  %Post{
    _id: #BSON.ObjectId<6327a7099626f7f61607e179>,
    inserted_at: ~U[2022-09-18 23:17:29.087092Z],
    title: "lorem ipsum dolor sit amet",
    updated_at: ~U[2022-09-18 23:17:29.087070Z]
  }

  iex(2)> Post.store(post)
  {:ok,
   %{
     _id: #BSON.ObjectId<6327a7099626f7f61607e179>,
     inserted_at: ~U[2022-09-18 23:17:29.087092Z],
     title: "lorem ipsum dolor sit amet",
     updated_at: ~U[2022-09-18 23:19:24.516648Z]
   }}

Is possible to change the field names, like Ecto does, and also change the default behaviour:

  defmodule Comment do

    use Mongo.Collection

    collection "comments" do
      attribute :text, String.t()
      timestamps(inserted_at: :created, updated_at: :modified, default: &__MODULE__.truncated_date_time/0)
    end

    def new(text) do
      Map.put(new(), :text, text)
    end

    def truncated_date_time() do
      utc_now = DateTime.utc_now()
      DateTime.truncate(utc_now, :second)
    end
  end

  iex(1)> comment = Comment.new("useful comment")
  %Comment{
    _id: #BSON.ObjectId<6327aa979626f7f616ae5b4a>,
    created: ~U[2022-09-18 23:32:39Z],
    modified: ~U[2022-09-18 23:32:39Z],
    text: "useful comment"
  }

  iex(2)> {:ok, comment} = MyRepo.insert(comment)
  {:ok,
   %Comment{
     _id: #BSON.ObjectId<6327aa979626f7f616ae5b4a>,
     created: ~U[2022-09-18 23:32:39Z],
     modified: ~U[2022-09-18 23:32:42Z],
     text: "useful comment"
   }}

  iex(3)> {:ok, comment} = MyRepo.update(%{comment | text: "not so useful comment"})
  {:ok,
   %Comment{
     _id: #BSON.ObjectId<6327aa979626f7f616ae5b4a>,
     created: ~U[2022-09-18 23:32:39Z],
     modified: ~U[2022-09-18 23:32:46Z],
     text: not so useful comment"
   }}

The timestamps macro has some limitations as it does not run in batch commands like insert_all or update_all, nor does it update embedded documents.