Domo behaviour (Domo v1.5.7) View Source

A library to validate values of nested structs with their type spec t() and associated precondition functions.

Example apps

🔗 JSON parsing and validation example

🔗 Commanded + Domo combo used in Event Sourcing and CQRS app

🔗 Ecto + Domo combo in example_avialia app

🔗 TypedStruct + Domo combo in example_typed_integrations app

Description

Used in a struct's module, the library adds constructor, validation, and reflection functions. When called, constructor and validation functions guarantee the following:

  • A struct or a group of nested structs conforms to their t() types.
  • The struct's data consistently follows the business rules given by type-associated precondition functions.

If the conditions described above are not met, the constructor and validation functions return an error.

The business rule expressed via the precondition function can be shared across all structs referencing the appropriate type.

In terms of Domain Driven Design, types and associated precondition functions define the invariants relating structs to each other.

Tour

Let's say that we have a LineItem and PurchaseOrder structs with relating invariant that is the sum of line item amounts should be less then order's approved limit. That can be expressed like the following:

defmodule LineItem do
  use Domo

  defstruct amount: 0

  @type t :: %__MODULE__{amount: non_neg_integer()}
end

defmodule PurchaseOrder do
  use Domo

  defstruct id: 1000, approved_limit: 200, items: []

  @type id :: non_neg_integer()
  precond(id: &(1000 <= &1 and &1 <= 5000))

  @type t :: %__MODULE__{
          id: id(),
          approved_limit: pos_integer(),
          items: [LineItem.t()]
        }
  precond(t: &validate_invariants/1)

  defp validate_invariants(po) do
    amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()

    if amounts <= po.approved_limit do
      :ok
    else
      {:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
    end
  end
end

Then PurchaseOrder struct can be constructed consistently with functions generated by Domo like the following:

PurchaseOrder.new()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}

The constructor function takes any Enumerable as the input value:

{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})
{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}

It returns the descriptive keyword list if there is an error in input arguments. And it validates nested structs automatically:

PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])
{:error,
 [
   items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}.
    Expected the value matching the [%LineItem{}] type.
    Underlying errors:
       - The element at index 0 has value %LineItem{amount: -5} that is invalid.
       - Value of field :amount is invalid due to Invalid value -5 for field :amount
         of %LineItem{}. Expected the value matching the non_neg_integer() type.",
   id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the
    value matching the non_neg_integer() type. And a true value from
    the precondition function \"&(1000 <= &1 and &1 <= 5000)\"
    defined for PurchaseOrder.id() type."
 ]}

The returned errors are verbose and are intended for debugging purposes. See the Error messages for a user section below for more options.

The manually updated struct can be validated like the following:

po
|> Map.put(:items, [LineItem.new!(amount: 150)])
|> PurchaseOrder.ensure_type()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}

Domo returns the error if the precondition function attached to the t() type that validates invariants for the struct as a whole fails:

updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)
{:error, [t: "Sum of line item amounts should be <= to approved limit"]}

Getting the list of the required fields of the struct that have type other then nil or any is like that:

PurchaseOrder.required_fields()
[:approved_limit, :id, :items]

See the Callbacks section for more details about functions added to the struct.

Error messages for a user

It's possible to attach error messages to types with the precond macro to display them later to the user. To filter such kinds of messages, pass the maybe_filter_precond_errors: true option to Domo generated functions like that:

defmodule Book do
  use Domo

  defstruct [:title, :pages]

  @type title :: String.t()
  precond title: &(if String.length(&1) > 1, do: :ok, else: {:error, "Book title is required."})

  @type pages :: pos_integer()
  precond pages: &(if &1 > 2, do: :ok, else: {:error, "Book should have more then 3 pages. Given (#{&1})."})

  @type t :: %__MODULE__{title: nil | title(), pages: nil | pages()}
end

defmodule Shelf do
  use Domo

  defstruct books: []

  @type t :: %__MODULE__{books: [Book.t()]}
end

defmodule PublicLibrary do
  use Domo

  defstruct shelves: []

  @type t :: %__MODULE__{shelves: [Shelf.t()]}
end

library = struct!(PublicLibrary, %{shelves: [struct!(Shelf, %{books: [struct!(Book, %{title: "", pages: 1})]})]})

PublicLibrary.ensure_type(library, maybe_filter_precond_errors: true)
{:error,
 [
   shelves: [
     "Book title is required.",
     "Book should have more then 3 pages. Given (1)."
   ]
]}

That output contains only a flattened list of precondition error messages from the deeply nested structure.

Compile-time and Run-time validations

At the project's compile-time, Domo performs the following checks:

  • It automatically validates that the default values given with defstruct/1 conform to struct's type and fulfill preconditions (can be disabled, see __using__/1 for more details).

  • It ensures that the struct using Domo built with new(!)/1 function to be a default argument for a function or a default value for a struct's field matches its type and preconditions.

At run-time, Domo validates structs matching their t() types.

Domo compiles TypeEnsurer module from struct's t() type to do all kinds of validations. There is a generated function with pattern matchings and guards for each struct's field. Constructor and validation functions of the struct delegate the work to the appropriate TypeEnsurer module.

After the compilation, the flow of control of the nested StructA validation can look like the following:

+--------------------+
| PurchaseOrder      |     +---------------------------+
|                    |     | PurchaseOrder.TypeEnsurer |
| new(!)/1 ----------|--_  |                           |
| ensure_type(!)/1 --|-----|-> ensure_field_type/1     |
+--------------------+     |   private functions       |
                           +----|----------------------+
                                |
+-----------------------+       |
| LineItem              |       |  +-------------------------+
|                       |       |  | LineItem.TypeEnsurer    |
| new(!)/1              |       |  |                         |
| ensure_type(!)/1      |       +--|-> ensure_field_type/1   |
+-----------------------+          |   private functions     |
                                   +-------------------------+

In interactive mode (iex / livebook) Domo generates TypeEnsurer module dynamically as the last step of struct's module definition.

In mix compile mode Domo generates all TypeEnsurer modules after elixir compiler finishes its job. The generated code can be found in _build/MIX_ENV/domo_generated_code folder. However, that is for information purposes only. The following compilation will overwrite all changes there.

Depending types tracking

Let's suppose a structure field's type depends on a type defined in another module. When the latter type or its precondition changes, Domo recompiles the former module automatically to update its TypeEnsurer to keep the type validation up to date.

Domo tracks type-depending modules and touches appropriate files during compilation.

That works for any number of intermediate modules between the module defining the struct's field and the module defining the field's final type.

Integration with Ecto

Ecto schema changeset can be automatically validated to conform to t() type and associated preconditions. Then the changeset function can be like the following:

defmodule Customer do
  use Ecto.Schema
  use Domo, skip_defaults: true

  import Ecto.Changeset
  import Domo.Changeset

  schema "customers" do
    field :first_name, :string
    field :last_name, :string
    field :birth_date, :date

    timestamps()
  end

  @type t :: %__MODULE__{
          first_name: String.t(),
          last_name: String.t(),
          birth_date: Date.t()
        }

  def changeset(changeset, attrs) do
    changeset
    |> cast(attrs, typed_fields())
    |> validate_required(required_fields())
    |> validate_type()
  end

  # Domo adds typed_fields/0, required_fields/0 functions to the schema.
  # Domo.Changeset defines validate_type/1 function.
end

See typed_fields/0, required_fields/0, and Domo.Changeset module documentation for details.

See detailed example is in the example_avialia project.

Integration with libraries generating t() type for a struct

Domo is compatible with most libraries that generate t() type for a struct and an Ecto schema, f.e. typed_struct and typed_ecto_schema respectfully. Just use Domo in the module, and that's it.

An example is in the example_typed_integrations project.

TypedStruct's submodule generation with :module option currently is not supported.

Installation

To use Domo in a project, add the following line to mix.exs dependencies:

{:domo, "~> 1.5"}

And the following line to the project's mix.exs file:

compilers: [:domo_compiler] ++ Mix.compilers()

To exclude the generated TypeEnsurer modules from mix test --coverage add the following line, that works since Elixir v1.13, to the project's mix.exs:

test_coverage: [ignore_modules: [~r/\.TypeEnsurer$/]]

To avoid mix format putting extra parentheses around precond/1 macro call, add the following import to the .formatter.exs:

[
  import_deps: [:domo]
]

Phoenix hot-reload

To enable Phoenix hot-reload for struct's type ensurers built by Domo, update the compilers in the mix.exs file like the following:

compilers: [:domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

And add the following line to the endpoint's configuration in the config.exs file:

config :my_app, MyApp.Endpoint,
  reloadable_compilers: [:phoenix, :domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

Umbrella application

Add the Domo dependency and compilers config as mentioned in the section above to the mix.exs file for each app using Domo.

You may add the same compilers config line to the app itself and to the root umbrella's mix.exs to enable recompile command to work correctly for iex -S mix run in the root.

Configuration

The options listed below can be set globally in the configuration with config :domo, option: value. The value given with use Domo, option: value overrides the global setting.

  • skip_defaults - if set to true, disables the validation of default values given with defstruct/1 to conform to the t() type at compile time. Default is false.

  • name_of_new_function - the name of the constructor function added to the module. The raising error function name is generated automatically from the given one by adding trailing !. Defaults are new and new! appropriately.

  • unexpected_type_error_as_warning - if set to true, prints warning instead of throwing an error for field type mismatch in the raising functions. Default is false.

  • remote_types_as_any - keyword list of type lists by modules that should be treated as any(). F.e. [{ExternalModule, [:t, :name]}, {OtherModule, :t}] Default is nil.

Run the Application.put_env(:domo, :verbose_in_iex, true) to enable verbose messages from domo in Interactive Elixir console.

Performance 🐢

Library affects the project's full recompilation time almost insignificantly.

The compilation times for the business application with 38 structs (8 fields each on average) having 158 modules in total are the following:

Mode       Average (by 3 measurements)  Deviation
No Domo    14.826s                      11.92
With Domo  15.711s                      7.34

Comparison: 
No Domo    14.826s 
With Domo  15.711s - 1.06x slower

The library ensures the correctness of data types at run-time and that comes with the computation price.

For the Tweet struct having 13 fields, the validation takes 3x times longer and 2x more memory then creating the struct with possibly invalid data. And validation of the Tweet struct (13 fields) after nesting a User struct (18 fields) takes 6x times longer and 5x more memory than simple struct's altering. That's a linear growth depending on the number of fields in the nested struct.

It may seem plodding, and it may look like a non-performant to run in production. It's not that. Validation can be executed wisely at the critical check-points of the app where valuable. As a result, users get the application with correct states that are valid in many business contexts.

The run-time benchmark can be executed after cloning the repo with cd benchmark && mix benchmark.

Generating 3000 inputs, may take a while.
=========================================

Generated 3000 tweet inputs with summary approx. size of 1350KB.
Generated 3000 user inputs with summary approx. size of 1287KB.
=========================================

Benchmark struct's construction
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.13.1
Erlang 24.1.5

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 8 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking __MODULE__.new!(map)...
Benchmarking struct!(__MODULE__, map)...

Name                               ips        average  deviation         median         99th %
struct!(__MODULE__, map)        237.81        4.21 ms    ±11.18%        4.18 ms        5.37 ms
__MODULE__.new!(map)             84.18       11.88 ms     ±5.75%       11.92 ms       13.31 ms

Comparison: 
struct!(__MODULE__, map)        237.81
__MODULE__.new!(map)             84.18 - 2.83x slower +7.67 ms

Memory usage statistics:

Name                             average  deviation         median         99th %
struct!(__MODULE__, map)         7.84 MB     ±0.09%        7.83 MB        7.84 MB
__MODULE__.new!(map)            15.64 MB     ±0.02%       15.63 MB       15.64 MB

Comparison: 
struct!(__MODULE__, map)         7.83 MB
__MODULE__.new!(map)            15.64 MB - 2.00x memory usage +7.80 MB

Benchmark struct's field modification
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.13.1
Erlang 24.1.5

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 8 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking struct!(tweet, user: user)...
Benchmarking struct!(tweet, user: user) |> __MODULE__.ensure_type!()...

Name                                                              ips        average  deviation         median         99th %
struct!(tweet, user: user)                                     407.83        2.45 ms     ±9.65%        2.42 ms        3.03 ms
struct!(tweet, user: user) |> __MODULE__.ensure_type!()         63.98       15.63 ms     ±4.86%       15.29 ms       17.59 ms

Comparison: 
struct!(tweet, user: user)                                     407.83
struct!(tweet, user: user) |> __MODULE__.ensure_type!()         63.98 - 6.37x slower +13.18 ms

Memory usage statistics:

Name                                                            average  deviation         median         99th %
struct!(tweet, user: user)                                      2.98 MB     ±0.12%        2.98 MB        2.98 MB
struct!(tweet, user: user) |> __MODULE__.ensure_type!()        14.59 MB     ±0.09%       14.59 MB       14.60 MB

Comparison: 
struct!(tweet, user: user)                                      2.98 MB
struct!(tweet, user: user) |> __MODULE__.ensure_type!()        14.59 MB - 4.90x memory usage +11.61 MB

Limitations

Parametrized types are not supported because it adds lots of complexity. Library returns {:type_not_found, :key} error for @type dict(key, value) :: [{key, value}] definition. Library returns error for type referencing parametrized type like @type field :: container(integer()).

Primitive types referencing themselves are not supported. Library returns an error for @type leaf :: leaf | nil definition. On the other hand, structs referencing themselves are supported. The library will build TypeEnsurer for the following definition @type t :: %__MODULE__{leaf: t | nil} and validate.

Migration

To migrate to a new version of Domo, please, clean and recompile the project with mix clean --deps && mix compile command.

Adoption

It's possible to adopt Domo library in the project having user-defined constructor functions named new/1 that interferes with Domo generated function name. Here's how:

  1. Add :domo dependency to the project, configure compilers as described in the installation section
  2. Set the name of the Domo generated constructor function by adding config :domo, :name_of_new_function, :constructor_name option into the confix.exs file, to prevent conflict with user-defined constructor function name
  3. Add use Domo to existing struct, f.e. FirstStruct
  4. Change all struct building calls to be done with Domo generated function with the name set on step 3 f.e. FistStruct.constructor_name(%{...})
  5. Remove user-defined constructor function new/1
  6. Repeat for each struct in the project

Link to this section Summary

Callbacks

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Creates a struct validating type conformance and preconditions.

Creates a struct validating type conformance and preconditions.

Returns the list of struct's fields having type others then nil or any().

Returns the list of struct's fields defined with explicit types in its t() type spec.

Functions

Uses Domo in the current struct's module to add constructor, validation, and reflection functions.

Checks whether the TypeEnsurer module exists for the given struct module.

Defines a precondition function for a field's type or the struct's type.

Link to this section Callbacks

Specs

ensure_type(struct :: struct()) :: {:ok, struct()} | {:error, any()}
Link to this callback

ensure_type(struct, opts)

View Source

Specs

ensure_type(struct :: struct(), opts :: keyword()) ::
  {:ok, struct()} | {:error, any()}

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid in the shape of {:ok, struct}. Otherwise returns the error in the shape of {:error, message_by_field}.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

Specs

ensure_type!(struct :: struct()) :: struct()

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid. Raises an ArgumentError otherwise.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

Specs

new() :: {:ok, struct()} | {:error, any()}

Specs

new(enumerable :: Enumerable.t()) :: {:ok, struct()} | {:error, any()}

Specs

new(enumerable :: Enumerable.t(), opts :: keyword()) ::
  {:ok, struct()} | {:error, any()}

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable in the shape of {:ok, struct_value}. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

If conditions described above are not fulfilled, the function returns an appropriate error in the shape of {:error, message_by_field}. message_by_field is a keyword list where the key is the name of the field and value is the string with the error message.

Keys in the enumerable that don't exist in the struct are automatically discarded.

Options

  • maybe_filter_precond_errors - when set to true, the values in message_by_field instead of string become a list of error messages from precondition functions. If there are no error messages from precondition functions for a field's type, then all errors are returned unfiltered. Helpful in taking one of the custom errors after executing precondition functions in a deeply nested type to communicate back to the user. F.e. when the field's type is another struct. Default is false.

Specs

new!() :: struct()

Specs

new!(enumerable :: Enumerable.t()) :: struct()

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

Raises an ArgumentError if conditions described above are not fulfilled.

This function will check if every given key-value belongs to the struct and raise KeyError otherwise.

Specs

required_fields() :: [atom()]

Specs

required_fields(opts :: keyword()) :: [atom()]

Returns the list of struct's fields having type others then nil or any().

Does not return meta fields with __underscored__ names.

Useful for validation of the required fields for emptiness. F.e. with validate_required/2 call in the Ecto changeset.

Options

  • :include_meta - when set to true, adds fields with __underscored__ names to the return list. Default is false.

Specs

typed_fields() :: [atom()]

Specs

typed_fields(opts :: keyword()) :: [atom()]

Returns the list of struct's fields defined with explicit types in its t() type spec.

Does not return meta fields with __underscored__ names and fields having any() type by default.

Includes fields that have nil type into the return list.

Options

  • :include_any_typed - when set to true, adds fields with any() type to the return list. Default is false.

  • :include_meta - when set to true, adds fields with __underscored__ names to the return list. Default is false.

Link to this section Functions

Link to this macro

__using__(opts)

View Source (macro)

Uses Domo in the current struct's module to add constructor, validation, and reflection functions.

defmodule Model do
  use Domo

  defstruct [:first_field, :second_field]
  @type t :: %__MODULE__{first_field: atom() | nil, second_field: any() | nil}

  # have added:
  # new!/1
  # new/2
  # ensure_type!/1
  # ensure_type/2
  # typed_fields/1
  # required_fields/1
end

use Domo can be called only within the struct module having t() type defined because it's used to generate __MODULE__.TypeEnsurer with validation functions for each field in the definition.

See details about t() type definition in Elixir TypeSpecs document.

The macro collects t() type definitions for the :domo_compiler which generates TypeEnsurer modules during the second pass of the compilation of the project. Generated validation functions rely on guards appropriate for the field types.

The generated code of each TypeEnsurer module can be found in _build/MIX_ENV/domo_generated_code folder. However, that is for information purposes only. The following compilation will overwrite all changes there.

The macro adds the following functions to the current module, that are the facade for the generated TypeEnsurer module: new!/1, new/2, ensure_type!/1, ensure_type/2, typed_fields/1, required_fields/1.

Options

The options listed below can be set globally in the configuration with config :domo, option: value. The value given with use Domo, option: value overrides the global setting.

  • skip_defaults - if set to true, disables the validation of default values given with defstruct/1 to conform to the t() type at compile time. Default is false.

  • name_of_new_function - the name of the constructor function added to the module. The raising error function name is generated automatically from the given one by adding trailing !. Defaults are new and new! appropriately.

  • unexpected_type_error_as_warning - if set to true, prints warning instead of throwing an error for field type mismatch in the raising functions. Default is false.

  • remote_types_as_any - keyword list of type lists by modules that should be treated as any(). F.e. [{ExternalModule, [:t, :name]}, {OtherModule, :t}] Default is nil.

Run the Application.put_env(:domo, :verbose_in_iex, true) to enable verbose messages from domo in Interactive Elixir console.

Link to this function

has_type_ensurer?(struct_module)

View Source

Checks whether the TypeEnsurer module exists for the given struct module.

Structs having TypeEnsurer can be validated with Domo generated callbacks.

Link to this macro

precond(type_fun)

View Source (macro)

Defines a precondition function for a field's type or the struct's type.

The type_fun argument is one element [type: fun] keyword list where type is the name of the type defined with the @type attribute and fun is a single argument user-defined precondition function.

The precondition function validates the value of the given type to match a specific format or to fulfil a set of invariants for the field's type or struct's type respectfully.

The macro should be called with a type in the same module where the @type definition is located. If that is no fulfilled, f.e., when the previously defined type has been renamed, the macro raises an ArgumentError.

defstruct [id: "I-000", amount: 0, limit: 15]

@type id :: String.t()
precond id: &validate_id/1

defp validate_id(id), do: match?(<<"I-", _::8*3>>, id)

@type t :: %__MODULE__{id: id(), amount: integer(), limit: integer()}
precond t: &validate_invariants/1

defp validate_invariants(s) do
  cond do
    s.amount >= s.limit ->
      {:error, "Amount #{s.amount} should be less then limit #{s.limit}."}

    true ->
      :ok
  end
end

TypeEnsurer module generated by Domo calls the precondition function with value of the valid type. Precondition function should return the following values: true | false | :ok | {:error, any()}.

In case of true or :ok return values TypeEnsurer finishes the validation of the field with ok. For the false return value, the TypeEnsurer generates an error message referencing the failed precondition function. And for the {:error, message} return value, it passes the message as one of the errors for the field value. message can be of any shape.

Macro adds the __precond__/2 function to the current module that routes a call to the user-defined function. The added function should be called only by Domo modules.

Attaching a precondition function to the type via this macro can be helpful to keep the same level of consistency across the domains modelled with structs sharing the given type.