View Source Unpickler (Unpickler v0.1.0)

Unpickler is a library for loading data in the Python's pickle format.

Supports all pickle protocols from 0 to 5.

Link to this section Summary

Functions

Loads the given pickle binary.

Link to this section Functions

Link to this function

load!(binary, opts \\ [])

View Source
@spec load!(
  binary(),
  keyword()
) :: {term(), rest :: binary()}

Loads the given pickle binary.

Basic literals and data structures are deserialized as corresponding Elixir terms whenever possible. The pickle data may include arbitrary Python objects, so in all other cases the Unpickler.Object struct is used. This struct holds all the information that would be used for object reconstruction. You can define a custom :object_resolver function to recognise certain objects and map them to whatever data structure you see fit.

object-references

Object references

Note that if the object hierarchy includes circular references, it is inherently impossible to represent in Elixir. One example of such data structure is a list with a reference to itself:

x = []
x.append(x)

On the other hand, multiple references to the same object are restored as expected, without duplicating memory, as in:

x = [1, 2, 3]
y = (x, x)

options

Options

  • :object_resolver - a function for constructing a custom term corresponding to a Python object. Receives Unpickler.Object as an argument and should return the term as {:ok, term} or :error if not applicable

  • :persistent_id_resolver - a function returning an object for the given persistent id. This function is required if the data includes a persistent id, otherwise an error is raised

examples

Examples

The scalar 1 would be loaded like so:

iex> Unpickler.load!(<<128, 4, 75, 1, 46>>)
{1, ""}

Next, a more complex data structure:

[1, 2.0, "text", (None, True), {"key": "val"}, b"\x01\x00"]
iex> data =
...>   <<128, 4, 149, 47, 0, 0, 0, 0, 0, 0, 0, 93, 148, 40, 75, 1, 71, 64, 0, 0, 0, 0, 0, 0, 0,
...>     140, 4, 116, 101, 120, 116, 148, 78, 136, 134, 148, 125, 148, 140, 3, 107, 101, 121,
...>     148, 140, 3, 118, 97, 108, 148, 115, 67, 2, 1, 0, 148, 101, 46>>
iex> Unpickler.load!(data)
{[1, 2.0, "text", {nil, true}, %{"key" => "val"}, <<1, 0>>], ""}

objects

Objects

Other objects end up as Unpickler.Object

from datetime import date
date.fromisoformat("2022-05-17")
iex> data =
...>   <<128, 4, 149, 32, 0, 0, 0, 0, 0, 0, 0, 140, 8, 100, 97, 116, 101, 116, 105, 109, 101,
...>     148, 140, 4, 100, 97, 116, 101, 148, 147, 148, 67, 4, 7, 230, 5, 17, 148, 133, 148, 82,
...>     148, 46>>
iex> Unpickler.load!(data)
{%Unpickler.Object{
   append_items: [],
   args: [<<7, 230, 5, 17>>],
   constructor: "datetime.date",
   kwargs: %{},
   set_items: [],
   state: nil
 }, ""}

For those, we can customize construction by specifying an :object_resolver

from datetime import date
date.fromisoformat("2022-05-17")
iex> data =
...>   <<128, 4, 149, 32, 0, 0, 0, 0, 0, 0, 0, 140, 8, 100, 97, 116, 101, 116, 105, 109, 101,
...>     148, 140, 4, 100, 97, 116, 101, 148, 147, 148, 67, 4, 7, 230, 5, 17, 148, 133, 148, 82,
...>     148, 46>>
iex> object_resolver = fn
...>   # See https://github.com/python/cpython/blob/3.10/Lib/datetime.py#L1094-L1105
...>   %{constructor: "datetime.date", args: [<<year_hi, year_lo, month, day>>]} ->
...>     {:ok, date} = Date.new(year_hi * 256 + year_lo, month, day)
...>     {:ok, date}
...>
...>   _ ->
...>     :error
...> end
iex> Unpickler.load!(data, object_resolver: object_resolver)
{~D[2022-05-17], ""}