py (erlang_python v2.3.1)
View SourceHigh-level API for executing Python code from Erlang.
This module provides a simple interface to call Python functions, execute Python code, and stream results from Python generators.
Examples
%% Call a Python function
{ok, Result} = py:call(json, dumps, [#{foo => bar}]).
%% Call with keyword arguments
{ok, Result} = py:call(json, dumps, [Data], #{indent => 2}).
%% Execute raw Python code
{ok, Result} = py:eval("1 + 2").
%% Stream from a generator
{ok, Stream} = py:stream(mymodule, generate_tokens, [Prompt]),
lists:foreach(fun(Token) -> io:format("~s", [Token]) end, Stream).Summary
Functions
Activate a Python virtual environment. This modifies sys.path to use packages from the specified venv. The venv path should be the root directory (containing bin/lib folders).
Wait for an async call to complete.
Wait for an async call with timeout. Note: Identical to await/2 - provided for API symmetry with async_call.
Call a Python async function (coroutine). Returns immediately with a reference. Use async_await/1,2 to get the result. This is for calling functions defined with async def in Python.
Call a Python async function with keyword arguments.
Execute multiple async calls concurrently using asyncio.gather. Takes a list of {Module, Func, Args} tuples and executes them all concurrently, returning when all are complete.
Stream results from a Python async generator. Returns a list of all yielded values.
Stream results from a Python async generator with kwargs.
Wait for an async call to complete.
Wait for an async call with timeout.
Call a Python function synchronously.
Call a Python function with keyword arguments or on a named pool.
Call a Python function with keyword arguments and optional timeout or pool.
Call a method on a Python object reference.
Fire-and-forget Python function call.
Fire-and-forget Python function call with context or kwargs.
Fire-and-forget Python function call with context and kwargs.
Clear all collected trace spans.
Configure Python logging to forward to Erlang logger. Uses default settings (debug level, default format).
Configure Python logging with options. Options: level => debug | info | warning | error (default: debug) format => string() - Python format string (optional)
Get the context for the current process.
Get a specific context by index.
Check if contexts have been started.
Deactivate the current virtual environment. Restores sys.path to its original state.
Disable distributed tracing.
Duplicate a file descriptor.
Enable distributed tracing from Python. After enabling, Python code can create spans with erlang.Span().
Ensure a virtual environment exists and activate it.
Ensure a virtual environment exists with options.
Evaluate a Python expression and return the result.
Evaluate a Python expression with local variables.
Evaluate a Python expression with local variables and timeout.
Execute Python statements (no return value expected).
Execute Python statements using a specific context.
Get the current execution mode. Returns one of: - free_threaded: Python 3.13+ with no GIL (Py_GIL_DISABLED) - subinterp: Python 3.12+ with per-interpreter GIL - multi_executor: Traditional Python with N executor threads
Force Python garbage collection. Performs a full collection (all generations). Returns the number of unreachable objects collected.
Force garbage collection of a specific generation. Generation 0 collects only the youngest objects. Generation 1 collects generations 0 and 1. Generation 2 (default) performs a full collection.
Get or create a process-local Python environment for a context.
Get all collected trace spans. Returns a list of span maps with keys: name, span_id, parent_id, start_time, end_time, duration_us, status, attributes, events
Get an attribute from a Python object reference.
Check if a term is a py_ref reference.
Get Python memory statistics. Returns a map containing: - gc_stats: List of per-generation GC statistics - gc_count: Tuple of object counts per generation - gc_threshold: Collection thresholds per generation - traced_memory_current: Current traced memory (if tracemalloc enabled) - traced_memory_peak: Peak traced memory (if tracemalloc enabled)
Get the number of executor threads. For multi_executor mode, this is the number of executor threads. For other modes, returns 1.
Execute multiple Python calls in true parallel using sub-interpreters. Each call runs in its own sub-interpreter with its own GIL, allowing CPU-bound Python code to run in parallel.
Register an Erlang function to be callable from Python. Python code can then call: erlang.call('name', arg1, arg2, ...) The function should accept a list of arguments and return a term.
Register an Erlang module:function to be callable from Python. The function will be called as Module:Function(Args).
Register a module or module/function to use a specific pool.
Reload a Python module across all contexts. This uses importlib.reload() to refresh the module from disk. Useful during development when Python code changes.
Delete a key from SharedDict.
Explicitly destroy a SharedDict.
Get a value from SharedDict with default undefined.
Get a value from SharedDict with custom default.
Get all keys from SharedDict.
Create a new process-scoped SharedDict.
Set a value in SharedDict.
Spawn a Python function call, returns immediately with a ref.
Spawn a Python function call with context or kwargs.
Spawn a Python function call with context and kwargs.
Start the process-per-context system with default settings.
Start the process-per-context system with options.
Clear all shared state.
Atomically decrement a counter by 1.
Atomically decrement a counter by Amount.
Fetch a value from shared state. This state is accessible from Python workers via state_get('key').
Atomically increment a counter by 1.
Atomically increment a counter by Amount.
Get all keys in shared state.
Remove a key from shared state.
Store a value in shared state. This state is accessible from Python workers via state_set('key', value).
Stop the process-per-context system.
Stream results from a Python generator. Returns a list of all yielded values.
Stream results from a Python generator with kwargs.
Cancel an active stream.
Stream results from a Python generator expression. Evaluates the expression and if it returns a generator, streams all values.
Stream results from a Python generator expression with local variables.
Start a true streaming iteration from a Python generator.
Start a true streaming iteration with options.
Async call - returns immediately with a reference. Use subinterp_await/1,2 to get the result. Worker uses erlang.send() to deliver result.
Wait for async call result.
Wait for async call result with timeout.
Call a function in a subinterpreter (blocking).
Call a function in a subinterpreter with kwargs (blocking).
Cast a call to subinterpreter (fire-and-forget, no result). Returns immediately. Use for side-effects where result is not needed.
Create an isolated subinterpreter with OWN_GIL. Returns a handle for making calls. The subinterpreter runs in a dedicated pthread with true parallelism.
Destroy a subinterpreter handle. Cleans up namespace, releases worker binding.
Evaluate expression in subinterpreter (blocking).
Evaluate expression with locals in subinterpreter (blocking).
Execute statements in subinterpreter (blocking, no return).
Check if the OWN_GIL thread pool is ready.
Start the OWN_GIL subinterpreter thread pool with default workers. Must be called before creating subinterpreter handles.
Start the OWN_GIL subinterpreter thread pool with N workers.
Get OWN_GIL thread pool statistics.
Stop the OWN_GIL subinterpreter thread pool.
Check if true parallel execution is supported. Returns true on Python 3.12+ which supports per-interpreter GIL.
Convert a Python object reference to an Erlang term.
Start memory allocation tracing. After starting, memory_stats() will include traced_memory_current and traced_memory_peak values.
Start memory tracing with specified frame depth. Higher frame counts provide more detailed tracebacks but use more memory.
Stop memory allocation tracing.
Unregister a previously registered function.
Unregister a module or module/function from pool routing.
Get information about the currently active virtual environment. Returns a map with venv_path and site_packages, or none if no venv is active.
Get Python version string.
Types
Functions
Activate a Python virtual environment. This modifies sys.path to use packages from the specified venv. The venv path should be the root directory (containing bin/lib folders).
.pth files in the venv's site-packages directory are processed, so editable installs created by uv, pip, or any PEP 517/660 compliant tool work correctly. New paths are inserted at the front of sys.path so that venv packages take priority over system packages.
Example:
ok = py:activate_venv(<<"/path/to/myenv">>).
{ok, _} = py:call(sentence_transformers, 'SentenceTransformer', [<<"all-MiniLM-L6-v2">>]).Wait for an async call to complete.
Wait for an async call with timeout. Note: Identical to await/2 - provided for API symmetry with async_call.
Call a Python async function (coroutine). Returns immediately with a reference. Use async_await/1,2 to get the result. This is for calling functions defined with async def in Python.
Example:
Ref = py:async_call(aiohttp, get, [<<"https://example.com">>]),
{ok, Response} = py:async_await(Ref).Call a Python async function with keyword arguments.
Execute multiple async calls concurrently using asyncio.gather. Takes a list of {Module, Func, Args} tuples and executes them all concurrently, returning when all are complete.
Example:
{ok, Results} = py:async_gather([
{aiohttp, get, [Url1]},
{aiohttp, get, [Url2]},
{aiohttp, get, [Url3]}
]).Stream results from a Python async generator. Returns a list of all yielded values.
Stream results from a Python async generator with kwargs.
Wait for an async call to complete.
Wait for an async call with timeout.
Call a Python function synchronously.
In worker mode, the call uses the process-local Python environment, allowing access to functions defined via py:exec() in the same process.
-spec call(pid(), py_module(), py_func(), py_args()) -> py_result(); (py_context_router:pool_name(), py_module(), py_func(), py_args()) -> py_result(); (py_module(), py_func(), py_args(), py_kwargs()) -> py_result().
Call a Python function with keyword arguments or on a named pool.
This function has multiple signatures: - call(Ctx, Module, Func, Args) - Call using a specific context pid - call(Pool, Module, Func, Args) - Call using a named pool (default, io, etc.) - call(Module, Func, Args, Kwargs) - Call with keyword arguments on default pool
In worker mode, calls use the process-local Python environment.
-spec call(pid(), py_module(), py_func(), py_args(), map()) -> py_result(); (py_context_router:pool_name(), py_module(), py_func(), py_args(), py_kwargs()) -> py_result(); (py_module(), py_func(), py_args(), py_kwargs(), timeout()) -> py_result().
Call a Python function with keyword arguments and optional timeout or pool.
This function has multiple signatures: - call(Ctx, Module, Func, Args, Opts) - Call using context with options map - call(Pool, Module, Func, Args, Kwargs) - Call using named pool with kwargs - call(Module, Func, Args, Kwargs, Timeout) - Call on default pool with timeout
Timeout is in milliseconds. Use infinity for no timeout. Rate limited via ETS-based semaphore to prevent overload.
Call a method on a Python object reference.
The reference carries the interpreter ID, so the call is automatically routed to the correct context.
Example:
{ok, Ref} = py:call(Ctx, builtins, list, [[1,2,3]], #{return => ref}),
{ok, 3} = py:call_method(Ref, '__len__', []).Fire-and-forget Python function call.
-spec cast(pid(), py_module(), py_func(), py_args()) -> ok; (py_module(), py_func(), py_args(), py_kwargs()) -> ok.
Fire-and-forget Python function call with context or kwargs.
Fire-and-forget Python function call with context and kwargs.
-spec clear_traces() -> ok.
Clear all collected trace spans.
-spec configure_logging() -> ok | {error, term()}.
Configure Python logging to forward to Erlang logger. Uses default settings (debug level, default format).
Configure Python logging with options. Options: level => debug | info | warning | error (default: debug) format => string() - Python format string (optional)
Example:
ok = py:configure_logging(#{level => info}).-spec context() -> pid().
Get the context for the current process.
If the process has a bound context (via bind_context/1), returns that. Otherwise, selects a context based on the current scheduler ID.
This provides automatic load distribution across contexts while maintaining scheduler affinity for cache locality.
-spec context(pos_integer()) -> pid().
Get a specific context by index.
-spec contexts_started() -> boolean().
Check if contexts have been started.
-spec deactivate_venv() -> ok | {error, term()}.
Deactivate the current virtual environment. Restores sys.path to its original state.
-spec disable_tracing() -> ok.
Disable distributed tracing.
Duplicate a file descriptor.
Creates a copy of an existing file descriptor. Use this when handing off a socket fd to Python while keeping Erlang's ability to close its socket.
Example:
{ok, ClientSock} = gen_tcp:accept(ListenSock),
{ok, Fd} = inet:getfd(ClientSock),
{ok, DupFd} = py:dup_fd(Fd),
py_reactor_context:handoff(DupFd, #{type => tcp}),
gen_tcp:close(ClientSock). %% Safe - Python has its own fd copy-spec enable_tracing() -> ok.
Enable distributed tracing from Python. After enabling, Python code can create spans with erlang.Span().
Ensure a virtual environment exists and activate it.
Creates a venv at Path if it doesn't exist, installs dependencies from RequirementsFile, and activates the venv.
RequirementsFile can be: - "requirements.txt" - standard pip requirements file - "pyproject.toml" - PEP 621 project file (installs with -e .)
Example:
ok = py:ensure_venv("priv/venv", "requirements.txt").Ensure a virtual environment exists with options.
Options: - {extras, [string()]} - Install optional dependencies (pyproject.toml) - {installer, uv | pip} - Package installer (default: auto-detect) - {python, string()} - Python executable for venv creation - force - Recreate venv even if it exists
Example:
%% With pyproject.toml and dev extras
ok = py:ensure_venv("priv/venv", "pyproject.toml", [
{extras, ["dev", "test"]}
]).
%% Force uv installer
ok = py:ensure_venv("priv/venv", "requirements.txt", [
{installer, uv}
]).Evaluate a Python expression and return the result.
In worker mode, evaluation uses the process-local Python environment. Variables defined via exec are visible in eval within the same process.
Evaluate a Python expression with local variables.
When the first argument is a pid (context), evaluates using the new process-per-context architecture with process-local environment.
-spec eval(pid(), string() | binary(), map()) -> py_result(); (string() | binary(), map(), timeout()) -> py_result().
Evaluate a Python expression with local variables and timeout.
When the first argument is a pid (context), evaluates using the new process-per-context architecture with process-local environment.
Timeout is in milliseconds. Use infinity for no timeout.
Execute Python statements (no return value expected).
In worker mode, the code runs in a process-local Python environment. Variables defined via exec persist within the calling Erlang process. In subinterpreter mode, each context has its own isolated namespace.
Execute Python statements using a specific context.
This is the explicit context variant of exec/1. Uses the process-local environment for the calling process.
-spec execution_mode() -> free_threaded | subinterp | multi_executor.
Get the current execution mode. Returns one of: - free_threaded: Python 3.13+ with no GIL (Py_GIL_DISABLED) - subinterp: Python 3.12+ with per-interpreter GIL - multi_executor: Traditional Python with N executor threads
Force Python garbage collection. Performs a full collection (all generations). Returns the number of unreachable objects collected.
Force garbage collection of a specific generation. Generation 0 collects only the youngest objects. Generation 1 collects generations 0 and 1. Generation 2 (default) performs a full collection.
Get or create a process-local Python environment for a context.
Each Erlang process can have Python environments per interpreter. The environments are stored in the process dictionary keyed by interpreter ID and are automatically freed when the process exits.
The environment is created inside the context's interpreter to ensure the correct memory allocator is used. This is critical for subinterpreters where each interpreter has its own memory allocator.
-spec get_traces() -> {ok, [map()]}.
Get all collected trace spans. Returns a list of span maps with keys: name, span_id, parent_id, start_time, end_time, duration_us, status, attributes, events
Get an attribute from a Python object reference.
Check if a term is a py_ref reference.
Get Python memory statistics. Returns a map containing: - gc_stats: List of per-generation GC statistics - gc_count: Tuple of object counts per generation - gc_threshold: Collection thresholds per generation - traced_memory_current: Current traced memory (if tracemalloc enabled) - traced_memory_peak: Peak traced memory (if tracemalloc enabled)
-spec num_executors() -> pos_integer().
Get the number of executor threads. For multi_executor mode, this is the number of executor threads. For other modes, returns 1.
Execute multiple Python calls in true parallel using sub-interpreters. Each call runs in its own sub-interpreter with its own GIL, allowing CPU-bound Python code to run in parallel.
Requires Python 3.12+. Use subinterp_supported/0 to check availability.
Example:
%% Run numpy matrix operations in parallel
{ok, Results} = py:parallel([
{numpy, dot, [MatrixA, MatrixB]},
{numpy, dot, [MatrixC, MatrixD]},
{numpy, dot, [MatrixE, MatrixF]}
]).On older Python versions, returns {error, subinterpreters_not_supported}.
Register an Erlang function to be callable from Python. Python code can then call: erlang.call('name', arg1, arg2, ...) The function should accept a list of arguments and return a term.
Register an Erlang module:function to be callable from Python. The function will be called as Module:Function(Args).
-spec register_pool(py_context_router:pool_name(), atom() | {atom(), atom()}) -> ok.
Register a module or module/function to use a specific pool.
After registration, calls to the module (or specific function) are automatically routed to the registered pool without changing call sites.
Examples:
%% Route all requests.* calls to io pool
py:register_pool(io, requests).
%% Route only aiohttp.get to io pool
py:register_pool(io, {aiohttp, get}).
%% Calls now automatically route to the correct pool
{ok, Resp} = py:call(requests, get, [Url]). %% -> io pool
{ok, 4.0} = py:call(math, sqrt, [16]). %% -> default poolReload a Python module across all contexts. This uses importlib.reload() to refresh the module from disk. Useful during development when Python code changes.
Note: This only affects already-imported modules. If the module hasn't been imported in a context yet, the reload is a no-op for that context.
Example:
%% After modifying mymodule.py on disk:
ok = py:reload(mymodule).Returns ok if reload succeeded in all contexts, or {error, Reasons} if any contexts failed.
Spawn a Python function call, returns immediately with a ref.
-spec spawn_call(pid(), py_module(), py_func(), py_args()) -> py_ref(); (py_module(), py_func(), py_args(), py_kwargs()) -> py_ref().
Spawn a Python function call with context or kwargs.
Spawn a Python function call with context and kwargs.
Start the process-per-context system with default settings.
Creates one context per scheduler using worker mode.
Start the process-per-context system with options.
Options: - contexts - Number of contexts to create (default: number of schedulers) - mode - Context mode: worker, subinterp, or owngil (default: worker)
-spec state_clear() -> ok.
Clear all shared state.
Atomically decrement a counter by 1.
Atomically decrement a counter by Amount.
Fetch a value from shared state. This state is accessible from Python workers via state_get('key').
Atomically increment a counter by 1.
Atomically increment a counter by Amount.
-spec state_keys() -> [term()].
Get all keys in shared state.
-spec state_remove(term()) -> ok.
Remove a key from shared state.
Store a value in shared state. This state is accessible from Python workers via state_set('key', value).
-spec stop_contexts() -> ok.
Stop the process-per-context system.
Stream results from a Python generator. Returns a list of all yielded values.
Stream results from a Python generator with kwargs.
-spec stream_cancel(reference()) -> ok.
Cancel an active stream.
Sends a cancellation signal to stop the stream iteration. Any pending values may still be delivered before the stream stops.
Stream results from a Python generator expression. Evaluates the expression and if it returns a generator, streams all values.
Stream results from a Python generator expression with local variables.
Start a true streaming iteration from a Python generator.
Unlike stream/3,4 which collects all values at once, this function returns immediately with a reference and sends values as events to the calling process as they are yielded.
Events sent to the owner process: - {py_stream, Ref, {data, Value}} - Each yielded value - {py_stream, Ref, done} - Stream completed - {py_stream, Ref, {error, Reason}} - Stream error
Supports both sync generators and async generators (coroutines).
Example:
{ok, Ref} = py:stream_start(builtins, iter, [[1,2,3,4,5]]),
receive_loop(Ref).
receive_loop(Ref) ->
receive
{py_stream, Ref, {data, Value}} ->
io:format("Got: ~p~n", [Value]),
receive_loop(Ref);
{py_stream, Ref, done} ->
io:format("Complete~n");
{py_stream, Ref, {error, Reason}} ->
io:format("Error: ~p~n", [Reason])
after 30000 ->
timeout
end.Start a true streaming iteration with options.
Options: - owner => pid() - Process to receive events (default: self())
Async call - returns immediately with a reference. Use subinterp_await/1,2 to get the result. Worker uses erlang.send() to deliver result.
Wait for async call result.
Wait for async call result with timeout.
-spec subinterp_call(reference(), py_module(), py_func(), py_args()) -> {ok, term()} | {error, term()}.
Call a function in a subinterpreter (blocking).
-spec subinterp_call(reference(), py_module(), py_func(), py_args(), py_kwargs()) -> {ok, term()} | {error, term()}.
Call a function in a subinterpreter with kwargs (blocking).
Cast a call to subinterpreter (fire-and-forget, no result). Returns immediately. Use for side-effects where result is not needed.
Create an isolated subinterpreter with OWN_GIL. Returns a handle for making calls. The subinterpreter runs in a dedicated pthread with true parallelism.
Requires the thread pool to be started first via subinterp_pool_start/0.
Example:
ok = py:subinterp_pool_start().
{ok, Sub} = py:subinterp_create().
{ok, Result} = py:subinterp_call(Sub, math, sqrt, [16.0]).
ok = py:subinterp_destroy(Sub).-spec subinterp_destroy(reference()) -> ok.
Destroy a subinterpreter handle. Cleans up namespace, releases worker binding.
Evaluate expression in subinterpreter (blocking).
Evaluate expression with locals in subinterpreter (blocking).
Execute statements in subinterpreter (blocking, no return).
-spec subinterp_pool_ready() -> boolean().
Check if the OWN_GIL thread pool is ready.
-spec subinterp_pool_start() -> ok | {error, term()}.
Start the OWN_GIL subinterpreter thread pool with default workers. Must be called before creating subinterpreter handles.
-spec subinterp_pool_start(non_neg_integer()) -> ok | {error, term()}.
Start the OWN_GIL subinterpreter thread pool with N workers.
-spec subinterp_pool_stats() -> map().
Get OWN_GIL thread pool statistics.
-spec subinterp_pool_stop() -> ok.
Stop the OWN_GIL subinterpreter thread pool.
-spec subinterp_supported() -> boolean().
Check if true parallel execution is supported. Returns true on Python 3.12+ which supports per-interpreter GIL.
Convert a Python object reference to an Erlang term.
-spec tracemalloc_start() -> ok | {error, term()}.
Start memory allocation tracing. After starting, memory_stats() will include traced_memory_current and traced_memory_peak values.
-spec tracemalloc_start(pos_integer()) -> ok | {error, term()}.
Start memory tracing with specified frame depth. Higher frame counts provide more detailed tracebacks but use more memory.
-spec tracemalloc_stop() -> ok | {error, term()}.
Stop memory allocation tracing.
Unregister a previously registered function.
Unregister a module or module/function from pool routing.
After unregistering, calls return to using the default pool.
Get information about the currently active virtual environment. Returns a map with venv_path and site_packages, or none if no venv is active.
Get Python version string.