View Source Pythonx (Pythonx v0.4.4)

Python interpreter embedded in Elixir.

Pythonx runs a Python interpreter in the same OS process as your Elixir application, allowing you to evaluate Python code and conveniently convert between Python and Elixir data structures.

The goal of this project is to better integrate Python workflows within Livebook and its usage in actual projects must be done with care due to Python's global interpreter lock (GIL), which prevents multiple threads from executing Python code at the same time. Consequently, calling Pythonx from multiple Elixir processes does not provide the concurrency you might expect and thus it can be a source of bottlenecks. However, this concerns regular Python code. Packages with CPU-intense functionality, such as numpy, have native implementation of many functions and invoking those releases the GIL. GIL is also released when waiting on I/O operations. In other words, if you are using this library to integrate with Python, make sure it happens in a single Elixir process or that its underlying libraries can deal with concurrent invocation. Otherwise, prefer to use Elixir's System.cmd/3 or Ports to manage multiple Python programs via I/O.

Usage (script)

Add Pythonx to your dependencies:

Mix.install([
  {:pythonx, "~> 0.4.0"}
])

Initialize the interpreter, specifying the desired Python version and dependencies:

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.2"
]
""")

Evaluate Python code:

{result, globals} =
  Pythonx.eval(
    """
    y = 10
    x + y
    """,
    %{"x" => 1}
  )

Pythonx.decode(result)
#=> 11

globals
#=> %{
#=>   "x" => #Pythonx.Object<
#=>     1
#=>   >,
#=>   "y" => #Pythonx.Object<
#=>     10
#=>   >
#=> }

In a dynamic evaluation environment, such as IEx and Livebook, you can also use the ~PY sigil:

import Pythonx

x = 1

~PY"""
y = 10
result = x + y
"""

result
#=> #Pythonx.Object<
#=>   11
#=> >

y
#=> #Pythonx.Object<
#=>   10
#=> >

Usage (application)

Add Pythonx to your dependencies:

def deps do
  [
    {:pythonx, "~> 0.4.0"}
  ]
end

Configure the desired Python version and dependencies in your config/config.exs:

import Config

config :pythonx, :uv_init,
  pyproject_toml: """
  [project]
  name = "project"
  version = "0.0.0"
  requires-python = "==3.13.*"
  dependencies = [
    "numpy==2.2.2"
  ]
  """

With that, you can use Pythonx.eval/2 and other APIs in your application. The downloads will happen at compile time, and the interpreter will get initialized automatically on boot. All necessary files are placed in Pythonx priv directory, so it is compatible with Elixir releases.

Note that currently the ~PY sigil does not work as part of Mix project code. This limitation is intentional, since in actual applications it is preferable to manage the Python globals explicitly.

How it works

CPython (the reference implementation of the Python programming language) provides a python executable that runs Python code, and that is the usual interface that developers use to interact with the interpreter. However, most the CPython functionality is also available as a dynamically linked library (.so, .dylib or .dll, depending on the platform). The python executable can be thought of as a program build on top of that library.

With this design, any C/C++ application can link the Python library and use its API to execute Python code and interact with Python objects on a low level. Taking this a step further, any language with C/C++ interoperability can interact with Python in the same manner. This usage of CPython is referred to as embedding Python.

Elixir provides C/C++ interoperability via Erlang NIFs and that is exactly how Pythonx embeds Python. As a result, the Python interpreter operates in the same OS process as the BEAM.

For more details refer to the official documentation on embedding Python.

Summary

Types

encoder()

Functions

decode(object)

Decodes a Python object to a term.

encode!(term, encoder \\ &Pythonx.Encoder.encode/2)

Encodes the given term to a Python object.

eval(code, globals, opts \\ [])

Evaluates the Python code.

sigil_PY(arg, list)

Convenience macro for Python code evaluation.

uv_init(pyproject_toml, opts \\ [])

Installs Python and dependencies using uv package manager and initializes the interpreter.

Types

encoder()

@type encoder() :: (term(), encoder() -> Pythonx.Object.t())

Functions

decode(object)

@spec decode(Pythonx.Object.t()) :: term()

Decodes a Python object to a term.

Converts the following Python types to the corresponding Elixir terms:

NoneType
bool
int
float
str
bytes
tuple
list
dict
set
frozenset

For all other types Pythonx.Object is returned.

Examples

iex> {result, %{}} = Pythonx.eval("(1, True, 'hello world')", %{})
iex> Pythonx.decode(result)
{1, true, "hello world"}

iex> {result, %{}} = Pythonx.eval("print", %{})
iex> Pythonx.decode(result)
#Pythonx.Object<
  <built-in function print>
>

encode!(term, encoder \\ &Pythonx.Encoder.encode/2)

@spec encode!(term(), encoder()) :: Pythonx.Object.t()

Encodes the given term to a Python object.

Encoding can be extended to support custom data structures, see Pythonx.Encoder.

Examples

iex> Pythonx.encode!({1, true, "hello world"})
#Pythonx.Object<
  (1, True, b'hello world')
>

eval(code, globals, opts \\ [])

@spec eval(String.t(), %{optional(String.t()) => term()}, keyword()) ::
  {Pythonx.Object.t() | nil, %{optional(String.t()) => Pythonx.Object.t()}}

Evaluates the Python code.

The globals argument is a map with global variables to be set for the evaluation. The map keys are strings, while the values can be any terms and they are automatically converted to Python objects by calling encode!/1.

The function returns the evaluation result and a map with the updated global variables. Note that the result is an object only if code ends with an expression, otherwise it is nil.

If the Python code raises an exception, Pythonx.Error is raised and the message includes the usual Python error message with traceback.

All writes to the Python standard output are sent to caller's group leader, while writes to the standard error are sent to the :standard_error process. Reading from the standard input is not supported and raises and error.

Concurrency

The Python interpreter has a mechanism known as global interpreter lock (GIL), which prevents from multiple threads executing Python code at the same time. Consequently, calling eval/2 from multiple Elixir processes does not provide the concurrency you might expect and thus it can be a source of bottlenecks. However, this concerns regular Python code. Packages with CPU-intense functionality, such as numpy, have native implementation of many functions and invoking those releases the GIL. GIL is also released when waiting on I/O operations.

Options

:stdout_device - IO process to send Python stdout output to. Defaults to the caller's group leader.
:stderr_device - IO process to send Python stderr output to. Defaults to the global :standard_error.

Examples

iex> {result, globals} =
...>   Pythonx.eval(
...>     """
...>     y = 10
...>     x + y
...>     """,
...>     %{"x" => 1}
...>   )
iex> result
#Pythonx.Object<
  11
>
iex> globals["x"]
#Pythonx.Object<
  1
>
iex> globals["y"]
#Pythonx.Object<
  10
>

You can carry evaluation state by passing globals from one evaluation to the next:

iex> {_result, globals} = Pythonx.eval("x = 1", %{})
iex> {result, _globals} = Pythonx.eval("x + 1", globals)
iex> result
#Pythonx.Object<
  2
>

Mutability

Reassigning variables will have no effect on the given globals, the returned globals will simply hold different objects:

iex> {_result, globals1} = Pythonx.eval("x = 1", %{})
iex> {_result, globals2} = Pythonx.eval("x = 2", globals1)
iex> globals1["x"]
#Pythonx.Object<
  1
>
iex> globals2["x"]
#Pythonx.Object<
  2
>

However, objects in globals are not automatically cloned, so if you explicitly mutate an object, it changes across all references:

iex> {_result, globals1} = Pythonx.eval("x = []", %{})
iex> {_result, globals2} = Pythonx.eval("x.append(1)", globals1)
iex> globals1["x"]
#Pythonx.Object<
  [1]
>
iex> globals2["x"]
#Pythonx.Object<
  [1]
>

sigil_PY(arg, list)

(macro)

Convenience macro for Python code evaluation.

This has all the characteristics of eval/2, except that globals are handled implicitly. This means that any Elixir variables referenced in the Python code will automatically get encoded and passed as globals for evaluation. Similarly, any globals assigned in the code will result in Elixir variables being defined.

Compilation

This macro evaluates Python code at compile time, so it requires the Python interpreter to be already initialized. In practice, this means that you can use this sigil in environments with dynamic evaluation, such as IEx and Livebook, but not in regular application code. In application code it is preferable to use eval/2 regardless, to make the globals management explicit.

Examples

iex> import Pythonx
iex> x = 1
iex> ~PY"""
...> y = 10
...> x + y
...> """
#Pythonx.Object<
  11
>
iex> x
1
iex> y
#Pythonx.Object<
  10
>

uv_init(pyproject_toml, opts \\ [])

@spec uv_init(
  String.t(),
  keyword()
) :: :ok

Installs Python and dependencies using uv package manager and initializes the interpreter.

The interpreter is automatically initialized using the installed Python. The dependency packages are added to the module search path.

Expects a string with pyproject.toml file content, which is used to configure the project environment. The config requires project.name and project.version fields to be set. It is also a good idea to specify the Python version by setting project.requires-python.

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
""")

To install Python packages, set the project.dependencies field:

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.2"
]
""")

For more configuration options, refer to the uv documentation.

Options

:force - if true, runs with empty project cache. Defaults to false.