Msgpack (msgpack_elixir v2.0.0)

An implementation of the MessagePack serialization format.

This module is the main entry point for the library, providing functions for encoding and decoding Elixir terms.

Quick Start

For the common case, you can encode an Elixir map or keyword list and decode it back. Note that by default, atoms used as map keys are encoded as strings.

iex> data = %{"id" => 1, "name" => "Elixir"}
iex> {:ok, encoded} = Msgpack.encode(data)
iex> Msgpack.decode(encoded)
{:ok, %{"id" => 1, "name" => "Elixir"}}

Capabilities

  • Type Support: Encodes and decodes common Elixir types, including integers, floats, binaries, lists, and maps.
  • Timestamp Extension: Automatically handles Elixir's NaiveDateTime and DateTime structs using the MessagePack Timestamp extension.
  • Custom Extensions: Provides MessagePack.Ext for working with custom MessagePack extension types.
  • Resource Limits: Includes options like :max_depth and :max_byte_size to limit resource allocation when decoding.
  • Telemetry Integration: Emits :telemetry events for monitoring and observability.
  • Extensible Structs: Allows any custom Elixir struct to be encoded by implementing the Msgpack.Encodable protocol.

Options

The behaviour of encode/2 and decode/2 can be customized by passing a keyword list of options. See the documentation for each function for a full description.

Common Encoding Options

  • :atoms - Controls how atoms are encoded (:string or :error).
  • :string_validation - Toggles UTF-8 validation for performance.

Common Decoding Options

  • :max_depth - Limits the nesting level for decoding collections.
  • :max_byte_size - Limits the memory allocation for large objects.

Summary

Functions

Decodes a MessagePack binary into an Elixir term.

Decodes a MessagePack binary, raising a Msgpack.DecodeError on failure.

Decodes a stream of MessagePack binaries into a stream of Elixir terms.

Encodes an Elixir term into a MessagePack binary.

Encodes an Elixir term into a MessagePack binary, raising an error on failure.

Encodes a stream of Elixir terms into a stream of MessagePack binaries.

Types

error_reason()

@type error_reason() ::
  {:unsupported_type, term()}
  | {:unsupported_atom, atom()}
  | :unexpected_eof
  | {:unknown_prefix, byte()}
  | {:trailing_bytes, binary()}
  | {:max_depth_reached, non_neg_integer()}
  | {:max_byte_size_exceeded, non_neg_integer()}
  | :invalid_timestamp

Functions

decode(binary, opts \\ [])

@spec decode(
  binary(),
  keyword()
) :: {:ok, term()} | {:error, error_reason()}

Decodes a MessagePack binary into an Elixir term.

Returns {:ok, term} on success, or {:error, reason} on failure.

Options

  • :max_depth - Sets a limit on the nesting level of arrays and maps to prevent stack exhaustion from maliciously crafted inputs. Defaults to 100.

  • :max_byte_size - Sets a limit on the declared byte size of any single string, binary, array, or map to prevent memory exhaustion attacks. Defaults to 10_000_000 (10MB).

Examples

Standard Decoding

For trusted inputs, you can decode directly without custom options.

iex> encoded = <<0x81, 0xA5, "hello", 0xA5, "world">>
iex> Msgpack.decode(encoded)
{:ok, %{"hello" => "world"}}

Securely Handling Untrusted Input

When decoding data from an external source, set limits to prevent denial-of-service attacks.

A deeply nested payload may exhaust the process stack:

iex> payload = <<0x91, 0x91, 0x91, 1>> # [[[1]]]
iex> Msgpack.decode(payload, max_depth: 2)
{:error, {:max_depth_reached, 2}}

A payload declaring a huge string can cause excessive memory allocation:

iex> payload = <<0xDB, 0xFFFFFFFF::32>> # A string of 4GB
iex> Msgpack.decode(payload, max_byte_size: 1_000_000)
{:error, {:max_byte_size_exceeded, 1_000_000}}

Detecting Malformed Data

The decoder will return an error tuple for malformed data, such as incomplete data or trailing bytes left over after a successful decode.

# A valid term followed by extra bytes
iex> Msgpack.decode(<<192, 42>>)
{:error, {:trailing_bytes, <<42>>}}

# Incomplete map data
iex> Msgpack.decode(<<0x81, 0xA3, "foo">>)
{:error, :unexpected_eof}

decode!(binary, opts \\ [])

@spec decode!(
  binary(),
  keyword()
) :: term()

Decodes a MessagePack binary, raising a Msgpack.DecodeError on failure.

This variant raises an exception on failure. It is intended for use when a decoding failure is considered an exceptional state, for example, when decoding data from a trusted internal service that is assumed to be well-formed.

For decoding data from external or untrusted sources where failure is a possible outcome, use decode/2 to handle the returned {:error, reason} tuple.

Options

Accepts the same options as decode/2.

Raises

  • Msgpack.DecodeError - If the binary is malformed, contains an unknown prefix, or has trailing bytes.

Examples

Basic success case:

iex> encoded = <<0x81, 0xA5, "hello", 0xA5, "world">>
iex> Msgpack.decode!(encoded)
%{"hello" => "world"}

Failure case:

iex> Msgpack.decode!(<<192, 42>>)
** (Msgpack.DecodeError) Failed to decode MessagePack binary. Reason = {:trailing_bytes, "*"}

decode_stream(enumerable, opts \\ [])

Decodes a stream of MessagePack binaries into a stream of Elixir terms.

This function provides a streaming, lazy interface for decoding, making it suitable for handling large datasets that do not fit into memory.

It delegates to Msgpack.StreamDecoder.decode/2.

For more detailed information on behavior, see the Msgpack.StreamDecoder module documentation.

Options

Accepts the same options as Msgpack.decode/2.

Examples

iex> objects = [1, "elixir", true]
iex> stream = Enum.map(objects, &Msgpack.encode!/1)
iex> Msgpack.decode_stream(stream) |> Enum.to_list()
[1, "elixir", true]

encode(term, opts \\ [])

@spec encode(
  term(),
  keyword()
) :: {:ok, binary()} | {:error, error_reason()}

Encodes an Elixir term into a MessagePack binary.

Returns {:ok, binary} on success, or {:error, reason} on failure.

Options

  • :atoms - Controls how atoms are encoded.

    • :string (default) - Encodes atoms as MessagePack strings.
    • :error - Returns an {:error, {:unsupported_atom, atom}} tuple if an atom is encountered.
  • :string_validation - Controls whether to perform UTF-8 validation on binaries.

    • true (default) - Validates binaries and encodes them as the str type if they are valid UTF-8, otherwise encodes them as the bin type. This ensures the output is compliant with the MessagePack specification's distinction between string and binary data, but has a performance cost.
    • false - Skips validation and encodes all binaries as the str type. This avoids the performance cost of validation but risks creating a payload with non-UTF-8 strings, which may be incompatible with other MessagePack decoders.
  • :deterministic - Controls whether map keys are sorted before encoding.

    • true (default) - Enables key sorting, which ensures that encoding the same map always produces the same binary.
    • false - Disables key sorting, which can provide a performance gain in cases where determinism is not required.

Custom Struct Support

This function can encode any custom Elixir struct that implements the Msgpack.Encodable protocol. This allows you to define custom serialization logic for your application structs.

For example, given a Product struct:

# 1. Define your struct
defmodule Product do
  defstruct [:id, :name]
end

# 2. Implement the protocol
defimpl Msgpack.Encodable, for: Product do
  def encode(%Product{id: id, name: name}) do
    # Transform the struct into an encodable term (e.g., a map)
    {:ok, %{"id" => id, "name" => name}}
  end
end

iex> product = %Product{id: 1, name: "Elixir"}
iex> {:ok, binary} = Msgpack.encode(product)
<<130, 162, 105, 100, 1, 164, 110, 97, 109, 101, 166, 69, 108, 105, 120, 105, 114>>

Examples

Standard Encoding

The default options encode atoms as strings, a common requirement when sending data between Elixir services.

iex> data = %{id: 1, name: "Elixir"}
iex> {:ok, encoded} = Msgpack.encode(data)
iex> Msgpack.decode(encoded)
{:ok, %{"id" => 1, "name" => "Elixir"}}

Strict Atom Handling

If you are interoperating with systems that do not have a concept of atoms, it is safer to disallow them completely during encoding.

iex> Msgpack.encode(%{name: "Elixir"}, atoms: :error)
{:error, {:unsupported_atom, :name}}

Encoding without String Validation (Unsafe)

For performance-critical paths where you can guarantee all binaries are valid UTF-8 strings, you can disable string validation.

iex> data = "What did the fish say when it swam into a wall? Dam!"
iex> {:ok, _} = Msgpack.encode(data, string_validation: false)

Encoding Raw Binary Data

If your data contains non-UTF-8 binary content (e.g., an image thumbnail), the default validator will encode it with the bin family type.

iex> Msgpack.encode(<<255, 128, 0>>)
{:ok, <<0xC4, 3, 255, 128, 0>>}

encode!(term, opts \\ [])

@spec encode!(
  term(),
  keyword()
) :: binary()

Encodes an Elixir term into a MessagePack binary, raising an error on failure.

This variant raises an exception on failure instead of returning an error tuple. It is intended for use in pipelines (|>) or in functions where an encoding failure is considered an exceptional event to be handled by try/rescue.

Options

Accepts the same options as encode/2.

Raises

Examples

iex> Msgpack.encode!(%{hello: "world"})
<<129, 165, 104, 101, 108, 108, 111, 165, 119, 111, 114, 108, 100>>

encode_stream(enumerable, opts \\ [])

Encodes a stream of Elixir terms into a stream of MessagePack binaries.

Each term in the input enumerable is encoded individually. The output stream will contain {:ok, binary} tuples for successful encodings or {:error, reason} tuples for failures.

This function delegates to Msgpack.StreamEncoder.encode/2.

Options

Accepts the same options as Msgpack.encode/2.

Examples

iex> terms = [1, "elixir", :world]
iex> Msgpack.encode_stream(terms, atoms: :string) |> Enum.to_list()
[
  {:ok, <<1>>},
  {:ok, <<166, 101, 108, 105, 120, 105, 114>>},
  {:ok, <<165, 119, 111, 114, 108, 100>>}
]