Flop (Flop v0.10.0) View Source

Flop is a helper library for filtering, ordering and pagination with Ecto.

Usage

The simplest way of using this library is just to use Flop.validate_and_run/3 and Flop.validate_and_run!/3. Both functions take a queryable and a parameter map, validate the parameters, run the query and return the query results and the meta information.

iex> Flop.Repo.insert_all(Flop.Pet, [
...>   %{name: "Harry", age: 4, species: "C. lupus"},
...>   %{name: "Maggie", age: 1, species: "O. cuniculus"},
...>   %{name: "Patty", age: 2, species: "C. aegagrus"}
...> ])
iex> params = %{order_by: ["name", "age"], page: 1, page_size: 2}
iex> {:ok, {results, meta}} =
...>   Flop.validate_and_run(
...>     Flop.Pet,
...>     params,
...>     repo: Flop.Repo
...>   )
iex> Enum.map(results, & &1.name)
["Harry", "Maggie"]
iex> meta.total_count
3
iex> meta.total_pages
2
iex> meta.has_next_page?
true

Under the hood, these functions just call Flop.validate/2 and Flop.run/3, which in turn calls Flop.all/3 and Flop.meta/3. If you need finer control about if and when to execute each step, you can call those functions directly.

See Flop.Meta for descriptions of the meta fields.

Global configuration

You can set some global options like the default Ecto repo via the application environment. All global options can be overridden by passing them directly to the functions or configuring the options for a schema module via Flop.Schema.

import Config

config :flop, repo: MyApp.Repo

See Flop.option/0 for a description of all available options.

Schema options

You can set some options for a schema by deriving Flop.Schema. The options are evaluated at the validation step.

defmodule Pet do
  use Ecto.Schema

  @derive {Flop.Schema,
           filterable: [:name, :species],
           sortable: [:name, :age],
           default_limit: 20,
           max_limit: 100}

  schema "pets" do
    field :name, :string
    field :age, :integer
    field :species, :string
    field :social_security_number, :string
  end
end

You need to pass the schema to Flop.validate/2 or any function that includes the validation step with the :for option.

iex> params = %{"order_by" => ["name", "age"], "limit" => 5}
iex> {:ok, flop} = Flop.validate(params, for: Flop.Pet)
iex> flop.limit
5

iex> params = %{"order_by" => ["name", "age"], "limit" => 200}
iex> {:error, changeset} = Flop.validate(params, for: Flop.Pet)
iex> [{:limit, {msg, _}}] = changeset.errors
iex> msg
"must be less than or equal to %{number}"

iex> params = %{"order_by" => ["name", "age"], "limit" => 200}
iex> {:error, changeset} =
...>   Flop.validate_and_run(
...>     Flop.Pet,
...>     params,
...>     for: Flop.Pet
...>   )
iex> [{:limit, {msg, _}}] = changeset.errors
iex> msg
"must be less than or equal to %{number}"

Ordering

To add an ordering clause to a query, you need to set the :order_by and optionally the :order_directions parameter. :order_by should be the list of fields, while :order_directions is a list of Flop.order_direction/0. :order_by and :order_directions are zipped when generating the ORDER BY clause. If no order directions are given, :asc is used as default.

iex> params = %{
...>   "order_by" => ["name", "age"],
...>   "order_directions" => ["asc", "desc"]
...> }
iex> {:ok, flop} = Flop.validate(params)
iex> flop.order_by
[:name, :age]
iex> flop.order_directions
[:asc, :desc]

Flop uses these two fields instead of a keyword list, so that the order instructions can be easily passed in a query string.

Pagination

For queries using OFFSET and LIMIT, you have the choice between page-based pagination parameters:

%{page: 5, page_size: 20}

and offset-based pagination parameters:

%{offset: 100, limit: 20}

For cursor-based pagination, you can either use :first/:after or :last/:before. You also need to pass the :order_by parameter or set a default order for the schema via Flop.Schema.

iex> Flop.Repo.insert_all(Flop.Pet, [
...>   %{name: "Harry", age: 4, species: "C. lupus"},
...>   %{name: "Maggie", age: 1, species: "O. cuniculus"},
...>   %{name: "Patty", age: 2, species: "C. aegagrus"}
...> ])
iex>
iex> # forward (first/after)
iex>
iex> params = %{first: 2, order_by: [:species, :name]}
iex> {:ok, {results, meta}} = Flop.validate_and_run(Flop.Pet, params)
iex> Enum.map(results, & &1.name)
["Patty", "Harry"]
iex> meta.has_next_page?
true
iex> end_cursor = meta.end_cursor
"g3QAAAACZAAEbmFtZW0AAAAFSGFycnlkAAdzcGVjaWVzbQAAAAhDLiBsdXB1cw=="
iex> params = %{first: 2, after: end_cursor, order_by: [:species, :name]}
iex> {:ok, {results, meta}} = Flop.validate_and_run(Flop.Pet, params)
iex> Enum.map(results, & &1.name)
["Maggie"]
iex> meta.has_next_page?
false
iex>
iex> # backward (last/before)
iex>
iex> params = %{last: 2, order_by: [:species, :name]}
iex> {:ok, {results, meta}} = Flop.validate_and_run(Flop.Pet, params)
iex> Enum.map(results, & &1.name)
["Harry", "Maggie"]
iex> meta.has_previous_page?
true
iex> start_cursor = meta.start_cursor
"g3QAAAACZAAEbmFtZW0AAAAFSGFycnlkAAdzcGVjaWVzbQAAAAhDLiBsdXB1cw=="
iex> params = %{last: 2, before: start_cursor, order_by: [:species, :name]}
iex> {:ok, {results, meta}} = Flop.validate_and_run(Flop.Pet, params)
iex> Enum.map(results, & &1.name)
["Patty"]
iex> meta.has_previous_page?
false

By default, it is assumed that the query result is a list of maps or structs. If your query returns a different data structure, you can pass the :get_cursor_value_func option to retrieve the cursor values. See Flop.option/0 and Flop.Cursor for more information.

You can restrict which pagination types are available. See Flop.option/0 for details.

Filters can be passed as a list of maps. It is recommended to define the filterable fields for a schema using Flop.Schema.

iex> Flop.Repo.insert_all(Flop.Pet, [
...>   %{name: "Harry", age: 4, species: "C. lupus"},
...>   %{name: "Maggie", age: 1, species: "O. cuniculus"},
...>   %{name: "Patty", age: 2, species: "C. aegagrus"}
...> ])
iex>
iex> params = %{filters: [%{field: :name, op: :=~, value: "Mag"}]}
iex> {:ok, {results, meta}} = Flop.validate_and_run(Flop.Pet, params)
iex> meta.total_count
1
iex> [pet] = results
iex> pet.name
"Maggie"

See Flop.Filter.op/0 for a list of all available filter operators.

GraphQL and Relay

The parameters used for cursor-based pagination follow the Relay specification, so you can just pass the arguments you get from the client on to Flop.

Flop.Relay can convert the query results returned by Flop.validate_and_run/3 into Edges and PageInfo formats required for Relay connections.

For example, if you have a context module like this:

defmodule MyApp.Flora
  import Ecto.query, warn: false

  alias MyApp.Flora.Plant

  def list_plants_by_continent(%Continent{} = continent, %{} = args) do
    Plant
    |> where(continent_id: ^continent.id)
    |> Flop.validate_and_run(args, for: Plant)
  end
end

Then your Absinthe resolver for the plants connection may look something like this:

def list_plants(args, %{source: %Continent{} = continent}) do
  with {:ok, result} <-
         Flora.list_plants_by_continent(continent, args) do
    {:ok, Flop.Relay.connection_from_result(result)}
  end
end

Link to this section Summary

Types

option()

Options that can be passed to most of the functions or that can be set via the application environment.

order_direction()

Represents the supported order direction values.

pagination_type()

Represents the pagination type.

t()

Represents the query parameters for filtering, ordering and pagination.

Functions

all(q, flop, opts \\ [])

Applies the given Flop to the given queryable and returns all matchings entries.

count(q, flop, opts \\ [])

Returns the total count of entries matching the filter conditions of the Flop.

filter(q, arg2)

Applies the filter parameter of a Flop.t/0 to an Ecto.Queryable.t/0.

meta(query_or_results, flop, opts \\ [])

Returns meta information for the given query and flop that can be used for building the pagination links.

order_by(q, flop)

Applies the order_by and order_directions parameters of a Flop.t/0 to an Ecto.Queryable.t/0.

paginate(q, arg2)

Applies the pagination parameters of a Flop.t/0 to an Ecto.Queryable.t/0.

push_order(flop, field)

Updates the order_by and order_directions values of a Flop struct.

query(q, flop)

Adds clauses for filtering, ordering and pagination to a Ecto.Queryable.t/0.

run(q, flop, opts \\ [])

Applies the given Flop to the given queryable, retrieves the data and the meta data.

validate(flop, opts \\ [])

Validates a Flop.t/0.

validate!(flop, opts \\ [])

Same as Flop.validate/2, but raises an Ecto.InvalidChangesetError if the parameters are invalid.

validate_and_run(q, flop, opts \\ [])

Validates the given flop parameters and retrieves the data and meta data on success.

validate_and_run!(q, flop, opts \\ [])

Same as Flop.validate_and_run/3, but raises on error.

Link to this section Types

option()

Specs

option() ::
  {:for, module()}
  | {:default_limit, pos_integer()}
  | {:filtering, boolean()}
  | {:get_cursor_value_func, (any(), [atom()] -> map())}
  | {:max_limit, pos_integer()}
  | {:ordering, boolean()}
  | {:pagination_types, [pagination_type()]}
  | {:repo, module()}

Options that can be passed to most of the functions or that can be set via the application environment.

:for - The schema module to be used for validation. Flop.Schema must be derived for the given module. This option is optional and can not be set globally. If it is not set, schema specific validation will be omitted. Used by the validation functions and passed on by any function calling a validation function.
:default_limit - Sets a global default limit for queries that is used if no default limit is set for a schema and no limit is set in the parameters. Can only be set in the application configuration.
:filtering (boolean) - Can be set to false to silently ignore filter parameters.
:get_cursor_value_func - 2-arity function used to get the (unencoded) cursor value from a record. Only used with cursor-based pagination. The first argument is the record, the second argument is the list of fields used in the ORDER BY clause. Needs to return a map with the order fields as keys and the the record values of these fields as values. Defaults to Flop.Cursor.get_cursor_from_map/2.
:max_limit - Sets a global maximum limit for queries that is used if no maximum limit is set for a schema. Can only be set in the application configuration.
:pagination_types - Defines which pagination types are allowed. Passing parameters for other pagination types will result in a validation error. By default, all pagination types are allowed. See also Flop.pagination_type/0. Note that an offset value of 0 and a limit are still accepted even if offset-based pagination is disabled.
:ordering (boolean) - Can be set to false to silently ignore order parameters. Default orders are still applied.
:repo - The Ecto Repo module to use for the database query. Used by all functions that execute a database query.

All options can be passed directly to the functions. Some of the options can be set on a schema level via Flop.Schema.

All options except :for can be set globally via the application environment.

import Config

config :flop,
  default_limit: 25,
  filtering: false,
  get_cursor_value_func: &MyApp.Repo.get_cursor_value/2,
  max_limit: 100,
  ordering: false,
  pagination_types: [:first, :last, :page],
  repo: MyApp.Repo

The look up order is:

option passed to function
option set for schema using Flop.Schema (only :max_limit, :default_limit and :pagination_types)
option set in global config (except :for)
default value (only :get_cursor_value_func)

order_direction()

Specs

order_direction() ::
  :asc
  | :asc_nulls_first
  | :asc_nulls_last
  | :desc
  | :desc_nulls_first
  | :desc_nulls_last

Represents the supported order direction values.

pagination_type()

Specs

pagination_type() :: :offset | :page | :first | :last

Represents the pagination type.

:offset - pagination using the offset and limit parameters
:page - pagination using the page and page_size parameters
:first - cursor-based pagination using the first and after parameters
:last - cursor-based pagination using the last and before parameters

t()

Specs

t() :: %Flop{
  after: String.t() | nil,
  before: String.t() | nil,
  filters: [Flop.Filter.t()] | nil,
  first: pos_integer() | nil,
  last: pos_integer() | nil,
  limit: pos_integer() | nil,
  offset: non_neg_integer() | nil,
  order_by: [atom() | String.t()] | nil,
  order_directions: [order_direction()] | nil,
  page: pos_integer() | nil,
  page_size: pos_integer() | nil
}

Represents the query parameters for filtering, ordering and pagination.

Fields

after: Used for cursor-based pagination. Must be used with first or a default limit.
before: Used for cursor-based pagination. Must be used with last or a default limit.
limit, offset: Used for offset-based pagination.
first Used for cursor-based pagination. Can be used alone to begin pagination or with after
last Used for cursor-based pagination.
page, page_size: Used for offset-based pagination as an alternative to offset and limit.
order_by: List of fields to order by. Fields can be restricted by deriving Flop.Schema in your Ecto schema.
order_directions: List of order directions applied to the fields defined in order_by. If empty or the list is shorter than the order_by list, :asc will be used as a default for each missing order direction.
filters: List of filters, see Flop.Filter.t/0.

Link to this section Functions

all(q, flop, opts \\ [])

(since 0.6.0)

Specs

all(Ecto.Queryable.t(), t(), [option()]) :: [any()]

Applies the given Flop to the given queryable and returns all matchings entries.

iex> Flop.all(Flop.Pet, %Flop{}, repo: Flop.Repo)
[]

You can also configure a default repo in your config files:

config :flop, repo: MyApp.Repo

This allows you to omit the third argument:

iex> Flop.all(Flop.Pet, %Flop{})
[]

Note that when using cursor-based pagination, the applied limit will be first + 1 or last + 1. The extra record is removed by Flop.run/3, but not by this function.

count(q, flop, opts \\ [])

(since 0.6.0)

Specs

count(Ecto.Queryable.t(), t(), [option()]) :: non_neg_integer()

Returns the total count of entries matching the filter conditions of the Flop.

The pagination and ordering option are disregarded.

iex> Flop.count(Flop.Pet, %Flop{}, repo: Flop.Repo)
0

You can also configure a default repo in your config files:

config :flop, repo: MyApp.Repo

This allows you to omit the third argument:

iex> Flop.count(Flop.Pet, %Flop{})
0

filter(q, arg2)

Specs

filter(Ecto.Queryable.t(), t()) :: Ecto.Queryable.t()

Applies the filter parameter of a Flop.t/0 to an Ecto.Queryable.t/0.

Used by Flop.query/2.

meta(query_or_results, flop, opts \\ [])

(since 0.6.0)

Specs

meta(Ecto.Queryable.t() | [any()], t(), [option()]) :: Flop.Meta.t()

Returns meta information for the given query and flop that can be used for building the pagination links.

iex> Flop.meta(Flop.Pet, %Flop{limit: 10}, repo: Flop.Repo)
%Flop.Meta{
  current_offset: 0,
  current_page: 1,
  end_cursor: nil,
  flop: %Flop{limit: 10},
  has_next_page?: false,
  has_previous_page?: false,
  next_offset: nil,
  next_page: nil,
  page_size: 10,
  previous_offset: nil,
  previous_page: nil,
  start_cursor: nil,
  total_count: 0,
  total_pages: 0
}

The function returns both the current offset and the current page, regardless of the pagination type. If the offset lies in between pages, the current page number is rounded up. This means that it is possible that the values for current_page and next_page can be identical. This can only occur if you use offset/limit based pagination with arbitrary offsets, but in that case, you will use the previous_offset, current_offset and next_offset values to render the pagination links anyway, so this shouldn't be a problem.

Unless cursor-based pagination is used, this function will run a query to figure get the total count of matching records.

order_by(q, flop)

Specs

order_by(Ecto.Queryable.t(), t()) :: Ecto.Queryable.t()

Applies the order_by and order_directions parameters of a Flop.t/0 to an Ecto.Queryable.t/0.

Used by Flop.query/2.

paginate(q, arg2)

Specs

paginate(Ecto.Queryable.t(), t()) :: Ecto.Queryable.t()

Applies the pagination parameters of a Flop.t/0 to an Ecto.Queryable.t/0.

The function supports both offset/limit based pagination and page/page_size based pagination.

If you validated the Flop.t/0 with Flop.validate/1 before, you can be sure that the given Flop.t/0 only has pagination parameters set for one pagination method. If you pass an unvalidated Flop.t/0 that has pagination parameters set for multiple pagination methods, this function will arbitrarily only apply one of the pagination methods.

Used by Flop.query/2.

push_order(flop, field)

(since 0.10.0)

Specs

push_order(t(), atom()) :: t()

Updates the order_by and order_directions values of a Flop struct.

If the field is not in the current order_by value, it will be prepended to the list. The order direction for the field will be set to :asc.
If the field is already at the front of the order_by list, the order direction will be reversed.
If the field is already in the list, but not at the front, it will be moved to the front and the order direction will be set to :asc.

Example

iex> flop = push_order(%Flop{}, :name)
iex> flop.order_by
[:name]
iex> flop.order_directions
[:asc]
iex> flop = push_order(flop, :age)
iex> flop.order_by
[:age, :name]
iex> flop.order_directions
[:asc, :asc]
iex> flop = push_order(flop, :age)
iex> flop.order_by
[:age, :name]
iex> flop.order_directions
[:desc, :asc]
iex> flop = push_order(flop, :species)
iex> flop.order_by
[:species, :age, :name]
iex> flop.order_directions
[:asc, :desc, :asc]
iex> flop = push_order(flop, :age)
iex> flop.order_by
[:age, :species, :name]
iex> flop.order_directions
[:asc, :asc, :asc]

query(q, flop)

Specs

query(Ecto.Queryable.t(), t()) :: Ecto.Queryable.t()

Adds clauses for filtering, ordering and pagination to a Ecto.Queryable.t/0.

The parameters are represented by the Flop.t/0 type. Any nil values will be ignored.

Examples

iex> flop = %Flop{limit: 10, offset: 19}
iex> Flop.query(Flop.Pet, flop)
#Ecto.Query<from p0 in Flop.Pet, limit: ^10, offset: ^19>

Or enhance an already defined query:

iex> require Ecto.Query
iex> flop = %Flop{limit: 10}
iex> Flop.Pet |> Ecto.Query.where(species: "dog") |> Flop.query(flop)
#Ecto.Query<from p0 in Flop.Pet, where: p0.species == "dog", limit: ^10>

Note that when using cursor-based pagination, the applied limit will be first + 1 or last + 1. The extra record is removed by Flop.run/3.

run(q, flop, opts \\ [])

(since 0.6.0)

Specs

run(Ecto.Queryable.t(), t(), [option()]) :: {[any()], Flop.Meta.t()}

Applies the given Flop to the given queryable, retrieves the data and the meta data.

This function does not validate the given flop parameters. You can validate the parameters with Flop.validate/2 or Flop.validate!/2, or you can use Flop.validate_and_run/3 or Flop.validate_and_run!/3 instead of this function.

iex> {data, meta} = Flop.run(Flop.Pet, %Flop{})
iex> data == []
true
iex> match?(%Flop.Meta{}, meta)
true

validate(flop, opts \\ [])

Specs

validate(t() | map(), [option()]) :: {:ok, t()} | {:error, Ecto.Changeset.t()}

Validates a Flop.t/0.

Examples

iex> params = %{"limit" => 10, "offset" => 0, "texture" => "fluffy"}
iex> Flop.validate(params)
{:ok,
 %Flop{
   filters: [],
   limit: 10,
   offset: 0,
   order_by: nil,
   order_directions: nil,
   page: nil,
   page_size: nil
 }}

iex> flop = %Flop{offset: -1}
iex> {:error, changeset} = Flop.validate(flop)
iex> changeset.valid?
false
iex> changeset.errors
[
  offset: {"must be greater than or equal to %{number}",
   [validation: :number, kind: :greater_than_or_equal_to, number: 0]}
]

It also makes sure that only one pagination method is used.

iex> params = %{limit: 10, offset: 0, page: 5, page_size: 10}
iex> {:error, changeset} = Flop.validate(params)
iex> changeset.valid?
false
iex> changeset.errors
[limit: {"cannot combine multiple pagination types", []}]

If you derived Flop.Schema in your Ecto schema to define the filterable and sortable fields, you can pass the module name to the function to validate that only allowed fields are used. The function will also apply any default values set for the schema.

iex> params = %{"order_by" => ["species"]}
iex> {:error, changeset} = Flop.validate(params, for: Flop.Pet)
iex> changeset.valid?
false
iex> [order_by: {msg, [_, {_, enum}]}] = changeset.errors
iex> msg
"has an invalid entry"
iex> enum
[:name, :age]

Note that currently, trying to use an existing field that is not allowed as seen above will result in the error message has an invalid entry, while trying to use a field name that does not exist in the schema (or more precisely: a field name that doesn't exist as an atom) will result in the error message is invalid. This might change in the future.

validate!(flop, opts \\ [])

(since 0.5.0)

Specs

validate!(t() | map(), [option()]) :: t()

Same as Flop.validate/2, but raises an Ecto.InvalidChangesetError if the parameters are invalid.

validate_and_run(q, flop, opts \\ [])

(since 0.6.0)

Specs

validate_and_run(Ecto.Queryable.t(), map() | t(), [option()]) ::
  {:ok, {[any()], Flop.Meta.t()}} | {:error, Ecto.Changeset.t()}

Validates the given flop parameters and retrieves the data and meta data on success.

iex> {:ok, {[], %Flop.Meta{}}} =
...>   Flop.validate_and_run(Flop.Pet, %Flop{}, for: Flop.Pet)
iex> {:error, %Ecto.Changeset{} = changeset} =
...>   Flop.validate_and_run(Flop.Pet, %Flop{limit: -1})
iex> changeset.errors
[
  limit: {"must be greater than %{number}",
    [validation: :number, kind: :greater_than, number: 0]}
]

Options

for: Passed to Flop.validate/2.
repo: The Ecto.Repo module. Required if no default repo is configured.
get_cursor_value_func: An arity-2 function to be used to retrieve an unencoded cursor value from a query result item and the order_by fields. Defaults to Flop.Cursor.get_cursor_from_map/2.

validate_and_run!(q, flop, opts \\ [])

(since 0.6.0)

Specs

validate_and_run!(Ecto.Queryable.t(), map() | t(), [option()]) ::
  {[any()], Flop.Meta.t()}

Same as Flop.validate_and_run/3, but raises on error.