View Source Pythonx (Pythonx v0.4.9)

Python interpreter embedded in Elixir.

Pythonx runs a Python interpreter in the same OS process as your Elixir application, allowing you to evaluate Python code and conveniently convert between Python and Elixir data structures.

The goal of this project is to better integrate Python workflows within Livebook and its usage in actual projects must be done with care due to Python's global interpreter lock (GIL), which prevents multiple threads from executing Python code at the same time. Consequently, calling Pythonx from multiple Elixir processes does not provide the concurrency you might expect and thus it can be a source of bottlenecks. However, this concerns regular Python code. Packages with CPU-intense functionality, such as numpy, have native implementation of many functions and invoking those releases the GIL. GIL is also released when waiting on I/O operations. In other words, if you are using this library to integrate with Python, make sure it happens in a single Elixir process or that its underlying libraries can deal with concurrent invocation. Otherwise, prefer to use Elixir's System.cmd/3 or Ports to manage multiple Python programs via I/O.

Usage (script)

Add Pythonx to your dependencies:

Mix.install([
  {:pythonx, "~> 0.4.0"}
])

Initialize the interpreter, specifying the desired Python version and dependencies:

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.2"
]
""")

Evaluate Python code:

{result, globals} =
  Pythonx.eval(
    """
    y = 10
    x + y
    """,
    %{"x" => 1}
  )

Pythonx.decode(result)
#=> 11

globals
#=> %{
#=>   "x" => #Pythonx.Object<
#=>     1
#=>   >,
#=>   "y" => #Pythonx.Object<
#=>     10
#=>   >
#=> }

In a dynamic evaluation environment, such as IEx and Livebook, you can also use the ~PY sigil:

import Pythonx

x = 1

~PY"""
y = 10
result = x + y
"""

result
#=> #Pythonx.Object<
#=>   11
#=> >

y
#=> #Pythonx.Object<
#=>   10
#=> >

Usage (application)

Add Pythonx to your dependencies:

def deps do
  [
    {:pythonx, "~> 0.4.0"}
  ]
end

Configure the desired Python version and dependencies in your config/config.exs:

import Config

config :pythonx, :uv_init,
  pyproject_toml: """
  [project]
  name = "project"
  version = "0.0.0"
  requires-python = "==3.13.*"
  dependencies = [
    "numpy==2.2.2"
  ]
  """

Additionally, you can configure a specific version of the uv package manager for Pythonx to use. This can impact the available Python versions.

import Config

config :pythonx, :uv_init,
  ...,
  uv_version: "0.7.21"

With that, you can use Pythonx.eval/2 and other APIs in your application. The downloads will happen at compile time, and the interpreter will get initialized automatically on boot. All necessary files are placed in Pythonx priv directory, so it is compatible with Elixir releases.

Note that currently the ~PY sigil does not work as part of Mix project code. This limitation is intentional, since in actual applications it is preferable to manage the Python globals explicitly.

Python API

Pythonx provides a Python module named pythonx with extra interoperability features.

pythonx.send_tagged_object(pid, tag, object)

Sends a Python object to an Elixir process identified by pid.

The Elixir process receives the message as a {tag, object} tuple, where tag is an atom and object is a Pythonx.Object struct.

Long-running evaluation

If you are sending messages from Python to Elixir, it likely means you have a long-running Python evaluation. If the evaluation holds onto GIL for long, you should make sure to only do it from a single Elixir process to avoid bottlenecks. For more details see the "Concurrency" notes in Pythonx.eval/3.

Decoding

The Elixir process receives a Pythonx.Object, which you may want to decode right away. Keep in mind that Pythonx.decode/1 requires GIL, so if the ongoing evaluation holds onto GIL for long, decoding itself may be blocked.

Parameters:

  • pid (pythonx.PID) – Opaque PID object, passed into the evaluation.
  • tag (str) – A tag appearning as atom in the Elixir message.
  • object (Any) – Any Python object to be sent as the message.

pythonx.PID

Opaque Python object that represents an Elixir PID.

This object cannot be created within Python, it needs to be passed into the evaluation as part of globals.

How it works

CPython (the reference implementation of the Python programming language) provides a python executable that runs Python code, and that is the usual interface that developers use to interact with the interpreter. However, most the CPython functionality is also available as a dynamically linked library (.so, .dylib or .dll, depending on the platform). The python executable can be thought of as a program build on top of that library.

With this design, any C/C++ application can link the Python library and use its API to execute Python code and interact with Python objects on a low level. Taking this a step further, any language with C/C++ interoperability can interact with Python in the same manner. This usage of CPython is referred to as embedding Python.

Elixir provides C/C++ interoperability via Erlang NIFs and that is exactly how Pythonx embeds Python. As a result, the Python interpreter operates in the same OS process as the BEAM.

For more details refer to the official documentation on embedding Python.

Summary

Types

encoder()

Functions

copy_remote_object(object)

Creates a local copy of a remote Pythonx.Object.

decode(object)

Decodes a Python object to a term.

encode!(term, encoder \\ &Pythonx.Encoder.encode/2)

Encodes the given term to a Python object.

eval(code, globals, opts \\ [])

Evaluates the Python code.

install_env()

Returns a map with opaque environment variables to initialize Pythonx in the same way as the current initialization.

install_paths()

Returns a list of paths that install_env/0 initialization depends on.

remote_eval(node, code, globals, opts \\ [])

Evaluates the Python code on node.

sigil_PY(arg, list)

Convenience macro for Python code evaluation.

uv_init(pyproject_toml, opts \\ [])

Installs Python and dependencies using uv package manager and initializes the interpreter.

Types

encoder()

@type encoder() :: (term(), encoder() -> Pythonx.Object.t())

Functions

copy_remote_object(object)

@spec copy_remote_object(Pythonx.Object.t()) :: Pythonx.Object.t()

Creates a local copy of a remote Pythonx.Object.

Remote objects can be returned when using remote_eval/4 or evaluating on a FLAME runner. This function makes a local copy of such objects, so that they can be passed to local eval/3, if desired.

If a local object is given, it is returned as is.

Pickling

The object are copied across nodes in a serialized format provided by the Python's build-in pickle module. While pickle supports basic Python types, and libraries implement custom pickling logic, certain Python values cannot be pickled by default (for example, local functions and lambdas). If you run into this limitation, you can add the cloudpickle package for extended pickling support:

cloudpickle==3.1.2

decode(object)

@spec decode(Pythonx.Object.t()) :: term()

Decodes a Python object to a term.

Converts the following Python types to the corresponding Elixir terms:

  • NoneType
  • bool
  • int
  • float
  • str
  • bytes
  • tuple
  • list
  • dict
  • set
  • frozenset
  • pythonx.PID

For all other types Pythonx.Object is returned.

Examples 

iex> {result, %{}} = Pythonx.eval("(1, True, 'hello world')", %{})
iex> Pythonx.decode(result)
{1, true, "hello world"}

iex> {result, %{}} = Pythonx.eval("print", %{})
iex> Pythonx.decode(result)
#Pythonx.Object<
  <built-in function print>
>

encode!(term, encoder \\ &Pythonx.Encoder.encode/2)

@spec encode!(term(), encoder()) :: Pythonx.Object.t()

Encodes the given term to a Python object.

Encoding can be extended to support custom data structures, see Pythonx.Encoder.

Examples 

iex> Pythonx.encode!({1, true, "hello world"})
#Pythonx.Object<
  (1, True, b'hello world')
>

eval(code, globals, opts \\ [])

@spec eval(String.t(), %{optional(String.t()) => term()}, keyword()) ::
  {Pythonx.Object.t() | nil, %{optional(String.t()) => Pythonx.Object.t()}}

Evaluates the Python code.

The globals argument is a map with global variables to be set for the evaluation. The map keys are strings, while the values can be any terms and they are automatically converted to Python objects by calling encode!/1.

The function returns the evaluation result and a map with the updated global variables. Note that the result is an object only if code ends with an expression, otherwise it is nil.

If the Python code raises an exception, Pythonx.Error is raised and the message includes the usual Python error message with traceback.

All writes to the Python standard output are sent to caller's group leader, while writes to the standard error are sent to the :standard_error process. Reading from the standard input is not supported and raises and error.

Concurrency

The Python interpreter has a mechanism known as global interpreter lock (GIL), which prevents from multiple threads executing Python code at the same time. Consequently, calling eval/2 from multiple Elixir processes does not provide the concurrency you might expect and thus it can be a source of bottlenecks. However, this concerns regular Python code. Packages with CPU-intense functionality, such as numpy, have native implementation of many functions and invoking those releases the GIL. GIL is also released when waiting on I/O operations.

Options

  • :stdout_device - IO process to send Python stdout output to. Defaults to the caller's group leader.

  • :stderr_device - IO process to send Python stderr output to. Defaults to the global :standard_error.

Examples 

iex> {result, globals} =
...>   Pythonx.eval(
...>     """
...>     y = 10
...>     x + y
...>     """,
...>     %{"x" => 1}
...>   )
iex> result
#Pythonx.Object<
  11
>
iex> globals["x"]
#Pythonx.Object<
  1
>
iex> globals["y"]
#Pythonx.Object<
  10
>

You can carry evaluation state by passing globals from one evaluation to the next:

iex> {_result, globals} = Pythonx.eval("x = 1", %{})
iex> {result, _globals} = Pythonx.eval("x + 1", globals)
iex> result
#Pythonx.Object<
  2
>

Mutability

Reassigning variables will have no effect on the given globals, the returned globals will simply hold different objects:

iex> {_result, globals1} = Pythonx.eval("x = 1", %{})
iex> {_result, globals2} = Pythonx.eval("x = 2", globals1)
iex> globals1["x"]
#Pythonx.Object<
  1
>
iex> globals2["x"]
#Pythonx.Object<
  2
>

However, objects in globals are not automatically cloned, so if you explicitly mutate an object, it changes across all references:

iex> {_result, globals1} = Pythonx.eval("x = []", %{})
iex> {_result, globals2} = Pythonx.eval("x.append(1)", globals1)
iex> globals1["x"]
#Pythonx.Object<
  [1]
>
iex> globals2["x"]
#Pythonx.Object<
  [1]
>

Remote execution

If you want to evaluate code on a remote node and get Pythonx.Object back, use remote_eval/4 instead of eval/3 to ensure proper lifetime of the Python objects.

Pythonx also integrates with FLAME. When you call eval/3 on a FLAME runner, enable the :track_resources option, so that the objects are properly tracked.

install_env()

@spec install_env() :: map()

Returns a map with opaque environment variables to initialize Pythonx in the same way as the current initialization.

When those environment variables are set, Pythonx is initialized on boot.

In particular, this can be used to make Pythonx initialize on FLAME nodes.

install_paths()

@spec install_paths() :: [String.t()]

Returns a list of paths that install_env/0 initialization depends on.

In particular, this can be used to make Pythonx initialize on FLAME nodes.

remote_eval(node, code, globals, opts \\ [])

@spec remote_eval(node(), String.t(), %{optional(String.t()) => term()}, keyword()) ::
  {Pythonx.Object.t() | nil, %{optional(String.t()) => Pythonx.Object.t()}}

Evaluates the Python code on node.

Any local Pythonx objects passed in globals will automatically be copied into the remote node. The returned result and globals are remote Pythonx objects, see the note below.

For more details and options, see eval/3.

Remote Pythonx objects

The result and globals returned by remote_eval/4 are Pythonx.Object structs, however those point to Python objects allocated on the remote node, where the evaluation run. It is guaranteed that those Python objects are kept alive, as long as you keep a reference to the local Pythonx.Object structs.

Avoid sending Pythonx.Object structs across nodes via regular messages or :erpc.call/4 invocations. When doing so, there is no guarantee that the corresponding Python objects are kept around.

Also note that calling eval/3 does not allow remote objects to be passed in globals. If you want to do that, you need to explicitly call copy_remote_object/1 first to get a local copy of the Python object.

sigil_PY(arg, list)

(macro)

Convenience macro for Python code evaluation.

This has all the characteristics of eval/2, except that globals are handled implicitly. This means that any Elixir variables referenced in the Python code will automatically get encoded and passed as globals for evaluation. Similarly, any globals assigned in the code will result in Elixir variables being defined.

Compilation

This macro evaluates Python code at compile time, so it requires the Python interpreter to be already initialized. In practice, this means that you can use this sigil in environments with dynamic evaluation, such as IEx and Livebook, but not in regular application code. In application code it is preferable to use eval/2 regardless, to make the globals management explicit.

Examples 

iex> import Pythonx
iex> x = 1
iex> ~PY"""
...> y = 10
...> x + y
...> """
#Pythonx.Object<
  11
>
iex> x
1
iex> y
#Pythonx.Object<
  10
>

uv_init(pyproject_toml, opts \\ [])

@spec uv_init(
  String.t(),
  keyword()
) :: :ok

Installs Python and dependencies using uv package manager and initializes the interpreter.

The interpreter is automatically initialized using the installed Python. The dependency packages are added to the module search path.

Expects a string with pyproject.toml file content, which is used to configure the project environment. The config requires project.name and project.version fields to be set. It is also a good idea to specify the Python version by setting project.requires-python.

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
""")

To install Python packages, set the project.dependencies field:

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.2"
]
""")

For more configuration options, refer to the uv documentation.

Environment variables

Python os.environ is initialized using the OS process environment. Consequently, it does not account for env vars modified with System.put_env/2 and similar. If you want to mirror certain env vars, you need to set them on os.environ using Pythonx.eval/2.

Free-threaded Python

Since Python 3.14, there is an official free-threaded Python build, which eliminates GIL and its caveats referred to throughout this documentation. To use a free-threaded build, you can specify an appropriate Python version variant using the :python option, for example Pythonx.uv_init(..., python: "3.14t").

Options

  • :force - if true, runs with empty project cache. Defaults to false.

  • :uv_version - select the version of the uv package manager to use. Defaults to "0.8.5".

  • :native_tls - if true, uses the system's native TLS implementation instead of vendored rustls. This is useful in corporate environments where the system certificate store must be used. Defaults to false.

  • :python - specifies the Python version to install. It is preferred to specify the version in pyproject_toml via the requires-python field. However, the :python option can be used to install a specific variant of Python, such as a free-threaded Python build, for example "3.14t".