Nestru (Nestru v1.0.1) View Source
A library to serialize between maps and nested structs.
Turns a map into a nested struct according to hints given to the library. And vice versa turns any nested struct into a map.
It works with maps/structs of any shape and level of nesting. Highly configurable
by implementing Nestru.Decoder
and Nestru.Encoder
protocols for structs.
Useful for translating map keys to struct's fields named differently. Or to specify default values missing in the map and required by struct.
The library's primary purpose is to serialize a map coming from a JSON payload or an Erlang term; at the same time, the map can be of any origin.
The input map can have atom or binary keys. The library takes the binary key first and then the same-named atom key if the binary key is missing while decoding the map. The library generates maps with atom keys during the struct encode operation.
Tour
Let's say we have an Order
with a total field which is an instance of a Total
struct.
And we want to serialize between an instance of Order
and a map.
Firstly, let's derive Nestru.Encoder
and Nestru.Decoder
protocols
and give a hint that the field :total
should hold a value of Total
struct
like the following:
defmodule Order do
@derive [Nestru.Encoder, {Nestru.Decoder, hint: %{total: Total}}]
defstruct [:id, :total]
end
defmodule Total do
@derive [Nestru.Encoder, Nestru.Decoder]
defstruct [:sum]
end
{:module, Total, <<70, 79, 82, 49, 0, 0, 8, ...>>, %Total{sum: nil}}
Secondly, we can encode the Order
into the map like that:
model = %Order{id: "A548", total: %Total{sum: 500}}
{:ok, map} = Nestru.encode(model)
{:ok, %{id: "A548", total: %{sum: 500}}}
And decode the map back into the Order
like the following:
map = %{
"id" => "A548",
"total" => %{"sum" => 500}
}
{:ok, model} = Nestru.decode(map, Order)
{:ok, %Order{id: "A548", total: %Total{sum: 500}}}
As you can see the data markup is in place, the Total
struct is nested within the Order
struct.
A list of structs in a field
Let's add the :items
field to Order1
struct to hold a list of LineItem
s
and give a hint to Nestru
on how to decode that field:
defmodule Order1 do
@derive {Nestru.Decoder, hint: %{total: Total, items: [LineItem]}}
defstruct [:id, :items, :total]
end
defmodule LineItem do
@derive Nestru.Decoder
defstruct [:amount]
end
{:module, LineItem, <<70, 79, 82, 49, 0, 0, 8, ...>>, %LineItem{amount: nil}}
Let's decode:
map = %{
"id" => "A548",
"items" => [%{"amount" => 150}, %{"amount" => 350}],
"total" => %{"sum" => 500}
}
{:ok, model} = Nestru.decode(map, Order1)
{:ok,
%Order1{
id: "A548",
items: [%LineItem{amount: 150}, %LineItem{amount: 350}],
total: %Total{sum: 500}
}}
Voilà, we have field values as nested structs 🎉
For the case when the list contains several structs of different types, please, see the Serializing type-dependent fields section below.
Date Time and URI
Let's say we have an Order2
struct with some URI
and DateTime
fields in it.
These attributes are structs in Elixir, at the same time they usually
kept as binary representations in a map.
Nestru
supports conversion between binaries
and structs, all we need to do is to implement the Nestry.Encoder
and Nestru.Decoder
protocols for these structs like the following:
# DateTime
defimpl Nestru.Encoder, for: DateTime do
def gather_fields_from_struct(struct, _context) do
{:ok, DateTime.to_string(struct)}
end
end
defimpl Nestru.Decoder, for: DateTime do
def decode_fields_hint(_empty_struct, _context, value) do
case DateTime.from_iso8601(value) do
{:ok, date_time, _offset} -> {:ok, date_time}
error -> error
end
end
end
# URI
defimpl Nestru.Encoder, for: URI do
def gather_fields_from_struct(struct, _context) do
{:ok, URI.to_string(struct)}
end
end
defimpl Nestru.Decoder, for: URI do
def decode_fields_hint(_empty_struct, _context, value) do
URI.new(value)
end
end
{:module, Nestru.Decoder.URI, <<70, 79, 82, 49, 0, 0, 8, ...>>, {:decode_fields_hint, 3}}
Order2
is defined like this:
defmodule Order2 do
@derive [Nestru.Encoder, {Nestru.Decoder, hint: %{date: DateTime, website: URI}}]
defstruct [:id, :date, :website]
end
{:module, Order2, <<70, 79, 82, 49, 0, 0, 8, ...>>, %Order2{id: nil, date: nil, website: nil}}
We can encode it to a map with binary fields like the following:
order = %Order2{id: "B445", date: ~U[2024-03-15 22:42:03Z], website: URI.parse("https://www.example.com/?book=branch")}
{:ok, map} = Nestru.encode(order)
{:ok, %{id: "B445", date: "2024-03-15 22:42:03Z", website: "https://www.example.com/?book=branch"}}
And decode it back:
Nestru.decode(map, Order2)
{:ok,
%Order2{
id: "B445",
date: ~U[2024-03-15 22:42:03Z],
website: %URI{
scheme: "https",
userinfo: nil,
host: "www.example.com",
port: 443,
path: "/",
query: "book=branch",
fragment: nil
}
}}
Error handling and path to the failed part of the map
Every implemented function of Nestru protocols can return {error, message}
tuple
in case of failure. When Nestru
receives the error tuple, it stops conversion
and bypasses the error to the caller.
defmodule Location do
@derive {Nestru.Decoder, hint: %{street: Street}}
defstruct [:street]
end
defmodule Street do
@derive {Nestru.Decoder, hint: %{house: House}}
defstruct [:house]
end
defmodule House do
defstruct [:number]
defimpl Nestru.Decoder do
def decode_fields_hint(_empty_struct, _context, value) do
if Nestru.has_key?(value, :number) do
{:ok, %{}}
else
{:error, "Can't continue without house number."}
end
end
end
end
So when we decode the following map missing the number
value, we will get
the error back:
map = %{
"street" => %{
"house" => %{
"name" => "Party house"
}
}
}
{:error, error} = Nestru.decode(map, Location)
{:error,
%{
get_in_keys: [#Function<8.67001686/3 in Access.key!/1>, #Function<8.67001686/3 in Access.key!/1>],
message: "Can't continue without house number.",
path: ["street", "house"]
}}
Nestru
wraps the error message into a map and adds path
and get_in_keys
fields to it. The path values point to the failed part of the map which can
be returned like the following:
get_in(map, error.get_in_keys)
%{"name" => "Party house"}
Maps with different key names
In some cases, the map's keys have slightly different names compared
to the target's struct field names. Fields that should be decoded into the struct
can be gathered by adopting Nestru.PreDecoder
protocol like the following:
defmodule Quote do
@derive [
{Nestru.PreDecoder, translate: %{"cost_value" => :cost}},
Nestru.Decoder
]
defstruct [:cost]
end
When we decode the map, Nestru
will put the value of the "cost_value"
key
for the :cost
key into the map and then complete the decoding:
map = %{
"cost_value" => 1280
}
Nestru.decode(map, Quote)
{:ok, %Quote{cost: 1280}}
For more sophisticated key mapping you can implement
the gather_fields_for_decoding/3
function of Nestru.PreDecoder
explicitly.
Serializing type-dependent fields
To convert a struct with a field that can have the value of multiple struct types into the map and back, the type of the field's value should be persisted. It's possible to do that like the following:
defmodule BookCollection do
defstruct [:name, :items]
defimpl Nestru.Encoder do
def gather_fields_from_struct(struct, _context) do
items_kinds =
Enum.map(struct.items, fn %module{} ->
module
|> Module.split()
|> Enum.join(".")
end)
{:ok, %{name: struct.name, items: struct.items, items_kinds: items_kinds}}
end
end
defimpl Nestru.Decoder do
def decode_fields_hint(_empty_struct, _context, value) do
items_kinds =
Enum.map(value.items_kinds, fn module_string ->
module_string
|> String.split(".")
|> Module.safe_concat()
end)
{:ok, %{items: &Nestru.decode_from_list(&1, items_kinds)}}
end
end
end
defmodule BookCollection.Book do
@derive [Nestru.Encoder, Nestru.Decoder]
defstruct [:title]
end
defmodule BookCollection.Magazine do
@derive [Nestru.Encoder, Nestru.Decoder]
defstruct [:issue]
end
Let's convert the nested struct into a map. The returned map gets
extra items_kinds
field with types information:
alias BookCollection.{Book, Magazine}
collection = %BookCollection{
name: "Duke of Norfolk's archive",
items: [
%Book{title: "The Spell in the Chasm"},
%Magazine{issue: "Strange Hunt"}
]
}
{:ok, map} = Nestru.encode(collection)
{:ok,
%{
items: [%{title: "The Spell in the Chasm"}, %{issue: "Strange Hunt"}],
items_kinds: ["BookCollection.Book", "BookCollection.Magazine"],
name: "Duke of Norfolk's archive"
}}
And restoring of the original nested struct is as simple as that:
{:ok, collection} = Nestru.decode(map, BookCollection)
{:ok,
%BookCollection{
items: [
%BookCollection.Book{title: "The Spell in the Chasm"},
%BookCollection.Magazine{issue: "Strange Hunt"}
],
name: "Duke of Norfolk's archive"
}}
Use with other libraries
Jason
JSON maps decoded with Jason library are supported with both binary and atoms keys.
ex_json_schema
ex_json_schema library can be used before decoding the input map with the JSON schema. To make sure that the structure of the input map is correct.
ExJSONPath
ExJsonPath library allows querying maps
(JSON objects) and lists (JSON arrays), using JSONPath expressions.
The queries can be useful in Nestru.PreDecoder.gather_fields_for_decoding/3
function to assemble fields for decoding from a map having a very different shape
from the target struct.
Domo
You can use the Domo library
to validate the t()
types of the nested struct values after
decoding with Nestru
.
Domo
can validate a nested struct in one pass, ensuring that
the struct's field values match its t()
type and associated preconditions.
Link to this section Summary
Functions
Decodes a map or a binary into the given struct.
Similar to decode/3
but checks if enforced struct's fields keys exist after decoding.
Decodes a list of values into the list of the given structs.
Similar to decode_from_list/2
but checks if enforced struct's fields keys
exist in the given maps.
Encodes the given struct into a map.
Similar to encode/1
.
Encodes the given list of structs into a list of values.
Gets the value for a specific key in map. Lookups a binary then an atom key.
Returns whether the given key exists in the given map as a binary or as an atom.
Link to this section Functions
Decodes a map or a binary into the given struct.
The first argument is a map having key-value pairs which supports both string and atom keys. Or a binary representation, f.e. date time in ISO 8601 format.
The second argument is a struct's module atom.
The third argument is a context value to be passed to implemented
functions of Nestru.PreDecoder
and Nestru.Decoder
protocols.
To give a hint on how to decode nested struct values or a list of such values
for the given field, implement Nestru.Decoder
protocol for the struct.
Function calls struct/2
to build the struct's value.
If given a map, keys that don't exist in the struct are automatically discarded.
Similar to decode/3
but checks if enforced struct's fields keys exist after decoding.
Returns a struct or raises an error.
Decodes a list of values into the list of the given structs.
The first argument is an input list.
If the second argument is a struct's module atom, then the function calls
the decode/3
on each input list item with the given module atom
as the second argument.
If the second argument is a list of struct module atoms, the function
calls the decode/3
function on each input list item with the module atom
taken at the same index of the second list as the second argument.
In this case, both lists should be of equal length.
The third argument is a context value to be passed to implemented
functions of Nestru.PreDecoder
and Nestru.Decoder
protocols.
The function returns a list of structs or the first error from decode/3
function.
Similar to decode_from_list/2
but checks if enforced struct's fields keys
exist in the given maps.
Returns a struct or raises an error.
Encodes the given struct into a map.
The first argument is a struct value to be encoded into map.
Encodes each field's value recursively when it is a struct or a list of structs.
The second argument is a context to be passed to Nestru.Encoder
protocol
function.
To insert additional fields or rename or drop existing ones before encoding
into map, implement Nestru.Encoder
protocol for the struct.
That can be used to keep additional type information for the field that can
have a value of various value types.
Similar to encode/1
.
Returns a map or raises an error.
Encodes the given list of structs into a list of values.
Calls encode/2
for each struct in the list. The Nestru.Encoder
protocol should be implemented for each struct module.
The function returns a list of values or the first error from encode/2
function.
Similar to encode_to_list/2
Returns list of maps or raises an error.
Gets the value for a specific key in map. Lookups a binary then an atom key.
If key is present in map then its value value is returned. Otherwise, default is returned.
If default is not provided, nil is used.
Returns whether the given key exists in the given map as a binary or as an atom.