SimpleSchema

SimpleSchema is a schema for validating JSON and storing it in a specific schema.

日本語ドキュメントはこちら

Motivation

When you are writing an HTTP API server, you may want to verify that the HTTP request body is correct. For validation only, you can use a library that implements JSON Schema. Fortunately Elixir has a library called ExJsonSchema that implements JSON Schema.

However, it is hard to write JSON Schema by hand. I want to use a more simple schema.

Also, JSON Schema only validates, so it takes time and effort to access the data.

json = Poison.decode!(conn.body_param)
:ok = validate(json)

hp = json["player"]["hp"]
# → I want to write json.player.hp

datetime = json["datetime"]                       # get a string and
{:ok, datetime, _} = DateTime.from_iso8601(value) # convert to DateTime
# → I want to get DateTime with json.datetime

I made a library called SimpleSchema to easily write the schema and convert the verified data.

How to Use

Use it as follows.

# Define a schema with defschema/1
defmodule Person do
  import SimpleSchema, only: [defschema: 1]

  defschema [
    name: :string,
    age: {:integer, minimum: 0},
  ]
end

# Map that decoded JSON string
json = %{
  "name" => "John Smith",
  "age" => 42,
}

# from_json!/2 with the map and Person, then a value is set in the Person structure
person = SimpleSchema.from_json!(Person, json)

assert person.name == "John Smith"
assert person.age == 42

If a passed JSON object is incorrect as the Person schema, you get an error like this:

bad_json = %{
  "name" => 100,             # not string
  "age" => -10,              # invalid age
  "__additional_key__" => 0, # an additional key
}

# from_json/2 fails
{:error, reason} = SimpleSchema.from_json(Person, bad_json)
IO.inspect reason

Output:

[{"Expected the value to be >= 0", "#/age"},
 {"Type mismatch. Expected String but got Integer.", "#/name"},
 {"Schema does not allow additional properties.", "#/__additional_key__"}]

This allows you to name common schemas and use them.

Simple Schema

I will explain SimpleSchema features in a bit more detail.

The schema which SimpleSchema library defines and which can be passed to the first argument of SimpleSchema.from_json/2 is called simple schema. For example, :integer is a simple schema.

value = SimpleSchema.from_json!(:integer, 10)
assert value == 10

:integer simple schema checks whether the passed value is an integer, and if it is an integer it will return that value. You can also possible to add restriction to integers.

value = SimpleSchema.from_json!({:integer, minimum: 10, maximum: 20}, 5)
# RuntimeError: [{"Expected the value to be >= 10", "#"}]

{:integer, opts} is also a simple schema. It checks whether the passed value is an integer and is within the range of 10 to 20, and if it is correct it will return that value.

% {...} is also a simple schema. And all each fields of that simple schema is also a simple schema.

schema = %{
  value: {:integer, optional: true},
  point: %{
    x: :integer,
    y: :integer,
  },
}
data = %{
  "point" => %{
    "x" => 10,
    "y" => 20,
  }
}
value = SimpleSchema.from_json!(schema, data)
# value == %{point: %{x: 10, y: 20}}
assert value.point.x == 10
assert value.point.y == 20

It checks whether the passed value is a map and is each fields of the passed value matches that simple schema. If it is correct, it will converts the key of the passed map to atom and it will return that value.

In addition, I added optional: true restriction to :value field. This can only be specified for children of the map. This mean “it will not cause an error even if this field is not present.” So SimpleSchema.from_json!/2 is successful even if "value" key does not exist in data.

List of Simpe Schema

A simple schema must be one of the following:

• :boolean or {:boolean, opts}
• :integer or {:integer, opts}
• :number or {:number, opts}
• :null or {:null, opts}
• :string or {:string, opts}
• :any or {:any, opts}
• %{...} or {%{...}, opts}
• [...] or {[...], opts}
• A module that implements SimpleSchema behaiviour, or {Module, opts}

opts specifies restrictions in the keyword list.

List of Restrictions

The list of restrictions is as follows.

• {:nullable, boolean}: If true, it can be set nil. It can be specified as any simple schema other than :null.
• {:minimum, integer}: Minimum value. It can be specified as :integer and :number.
• {:maximum, integer}: Maximum value. It can be specified as :integer and :number.
• {:min_items, non_neg_integer}: Minimum element count. It can be specified as :array.
• {:max_items, non_neg_integer}: Maximum element count. It can be specified as :array.
• {:min_length, non_neg_integer}: Minimum length. It can be specified as :string.
• {:max_length, non_neg_integer}: Maximum length. It can be specified as :string.
• {:enum, [...]}: List of possible values for elements. It can be specified as :integer and :string.
• {:format, :datetime | :email}: Validate by pre-defined format. It can be specified as :string.
• {:optional, boolean}: If true, the child element of %{...} is not required. It can be only specified as the child element of %{...}.
• {:tolerant, boolean}: If true, "additionalProperties" would be set to true, and will allow non-specified keys to be present in the child elements. It can be only specified as %{...}. Defaults to false.
• {:default, any}: If the default value is specified and a field of the map is not given, the specified default value is set to the field. It can be only specified as the child element of %{...}.
• {:field, string}: Corresponding JSON field name. It can be only specified as the child element of %{...}.

SimpleSchema behaiviour

A module that implements SimpleSchema behaiviour is also a simple schema. You can use this to name a specific schema or to convert it to a specific structure.

For example, to get the date according to ISO 8601 such as "2017-11-27T11:49:50+09:00" as a DateTime type, define it as follows.

defmodule DateTimeSchema do
  @behaviour SimpleSchema

  @impl SimpleSchema
  def schema(_opts) do
    {:string, format: :datetime}
  end

  @impl SimpleSchema
  def from_json(_schema, value, _opts) do
    case DateTime.from_iso8601(value) do
      {:ok, datetime, _} -> {:ok, datetime}
      {:error, reason} -> {:error, reason}
    end
  end

  @impl SimpleSchema
  def to_json(_schema, value, _opts) do
    {:ok, DateTime.to_iso8601(value)}
  end
end

Since DateTimeSchema is a simple schema, it can be passed to SimpleSchema.from_json!/2 as follows.

datetime = SimpleSchema.from_json!(DateTimeSchema, "2017-11-27T11:49:50+09:00")
# datetime == #DateTime<2017-11-27 02:49:50Z>

By implementing the SimpleSchema behavior like this, we can name DateTimeSchema to a specific schema and convert it to a structure of DateTime type. A module equivalent to DateTimeSchema above is already in SimpleSchema.Type.DateTime.

The functions required by SimpleSchema behavior are as follows.

@callback schema(opts :: Keyword.t) :: simple_schema
@callback from_json(schema :: simple_schema, json :: any, opts :: Keyword.t) :: {:ok, any} | {:error, any}
@callback to_json(schema :: simple_schema, value :: any, opts :: Keyword.t) :: {:ok, any} | {:error, any}

In schema/1, define the simple schema required by that module.

from_json/3 converts value to an arbitrary type and returns it. value has been verified with the simple schema returned by schema/1. For example value passed to DateTimeSchema.from_json/3 above has been verified by {:string, format: datetime}. So value is guaranteed that is a string and is :datetime format.

Note: If optimistic: true is specified in SimpleSchema.from_json/2, validation will not be done. In this case, the user is responsible for passing the correct value.

to_json/3 converts the passed value to a JSON value satisfying the simple schema. It performs the inverse conversion from from_json/3. This function is used inside SimpleSchema.to_json/2. You can return {:error, "not implemented"} if it is not necessary.

defschema/1

defschema/1 defines a structure by defstruct/1 and implements SimpleSchema behaviour.

defmodule Person do
  import SimpleSchema, only: [defschema: 1]

  defschema [
    name: :string,
    age: {:integer, minimum: 0},
  ]
end

This code is converted as follows.

defmodule Person do
  @enforce_keys [:name, :age]
  defstruct [:name, :age]

  @behaviour SimpleSchema

  @impl SimpleSchema
  def schema(_opts) do
    %{
      name: :string,
      age: {:integer, minimum: 0},
    }
  end

  @impl SimpleSchema
  def from_json(schema, value, _opts) do
    SimpleSchema.Type.json_to_struct(__MODULE__, schema, value)
  end

  @impl SimpleSchema
  def to_json(schema, value, _opts) do
    SimpleSchema.Type.struct_to_json(__MODULE__, schema, value)
  end
end

schema/1 defines a simple schema as a map with :name and :age. When calling SimpleSchema.from_json!/2, it verifies the passed JSON object, then calls Person.from_json/3 and converts value to the Person structure. Since we provides SimpleSchema.Type.json_to_struct/3 as a helper to convert JSON objects to specific structures, using this makes it easy to convert.

Examples

Simple usage:

iex> person_schema = %{name: :string, age: :integer}
iex>
iex> json = %{"name" => "John Smith", "age" => 42}
iex> SimpleSchema.from_json(person_schema, json)
{:ok, %{name: "John Smith", age: 42}}
iex>
iex> invalid_json = %{"name" => "John Smith", "age" => "invalid"}
iex> SimpleSchema.from_json(person_schema, invalid_json)
{:error, [{"Type mismatch. Expected Integer but got String.", "#/age"}]}

Name a simple schema:

defmodule Person do
  @behaviour SimpleSchema

  @impl SimpleSchema
  def schema(_opts) do
    %{
      name: :string,
      age: :integer,
    }
  end

  @impl SimpleSchema
  def from_json(schema, json) do
    SimpleSchema.Schema.from_json(schema, json)
  end

  @impl SimpleSchema
  def to_json(schema, json) do
    SimpleSchema.Schema.to_json(schema, json)
  end
end

iex> json = %{"name" => "John Smith", "age" => 42}
iex> SimpleSchema.from_json(Person, json)
{:ok, %{name: "John Smith", age: 42}}

# Nesting
defmodule Group do
  @behaviour SimpleSchema

  @impl SimpleSchema
  def schema(_opts) do
    %{
      name: :string,
      persons: [Person],
    }
  end

  @impl SimpleSchema
  def from_json(schema, json) do
    SimpleSchema.Schema.from_json(schema, json)
  end

  @impl SimpleSchema
  def to_json(schema, json) do
    SimpleSchema.Schema.to_json(schema, json)
  end
end

iex> json = %{"name" => "My Group",
...>          "persons" => [%{"name" => "John Smith", "age" => 42},
...>                        %{"name" => "Hans Schmidt", "age" => 18}]}
iex> SimpleSchema.from_json(Group, json)
{:ok, %{name: "My Group",
        persons: [%{name: "John Smith", age: 42},
                  %{name: "Hans Schmidt", age: 18}]}}

With struct:

defmodule StructPerson do
  import SimpleSchema, only: [defschema: 1]
  defschema [
    name: :string,
    age: :integer,
  ]
end

iex> json = %{"name" => "John Smith", "age" => 42}
iex> SimpleSchema.from_json(StructPerson, json)
{:ok, %StructPerson{name: "John Smith", age: 42}}

# Nesting
defmodule StructGroup do
  import SimpleSchema, only: [defschema: 1]
  defschema [
    name: :string,
    persons: [StructPerson],
  ]
end

iex> json = %{"name" => "My Group",
...>          "persons" => [%{"name" => "John Smith", "age" => 42},
...>                        %{"name" => "Hans Schmidt", "age" => 18}]}
iex> SimpleSchema.from_json(StructGroup, json)
{:ok, %StructGroup{name: "My Group",
                   persons: [%StructPerson{name: "John Smith", age: 42},
                             %StructPerson{name: "Hans Schmidt", age: 18}]}}

With restrictions:

defmodule StrictPerson do
  import SimpleSchema, only: [defschema: 1]
  defschema [
    name: {:string, min_length: 4},
    age: {:integer, minimum: 20, maximum: 65},
  ]
end

iex> json = %{"name" => "John Smith", "age" => 42}
iex> SimpleSchema.from_json(StrictPerson, json)
{:ok, %StrictPerson{name: "John Smith", age: 42}}

# Nesting
defmodule StrictGroup do
  import SimpleSchema, only: [defschema: 1]
  defschema [
    name: {:string, min_length: 4},
    persons: {[StrictPerson], min_items: 2},
  ]
end

iex> json = %{"name" => "My Group",
...>          "persons" => [%{"name" => "John Smith", "age" => 42},
...>                        %{"name" => "Hans Schmidt", "age" => 18}]}
iex> SimpleSchema.from_json(StrictGroup, json)
{:error, [{"Expected the value to be >= 20", "#/persons/1/age"}]}