Testing with Free-Threaded Python

View Source

This guide explains how to test erlang_python with Python's experimental free-threading mode (no GIL).

Overview

Python 3.13+ can be built with --disable-gil to create a "free-threaded" build where the Global Interpreter Lock is removed. This allows true parallel execution of Python code across multiple threads.

erlang_python automatically detects free-threaded Python and uses direct execution (no executor thread) for maximum performance.

Installing Free-Threaded Python

# Install pyenv if not already installed
curl https://pyenv.run | bash

# Install free-threaded Python 3.13+
PYTHON_CONFIGURE_OPTS="--disable-gil" pyenv install 3.13.0

# Or for 3.14+
PYTHON_CONFIGURE_OPTS="--disable-gil" pyenv install 3.14.0

Option 2: Build from Source

# Download Python source
wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tar.xz
tar xf Python-3.13.0.tar.xz
cd Python-3.13.0

# Configure with --disable-gil
./configure --prefix=$HOME/python-nogil --disable-gil --enable-optimizations

# Build and install
make -j$(nproc)
make install

Option 3: macOS with Homebrew

# Check if free-threading option is available
brew info python@3.13

# If available:
brew install python@3.13 --with-freethreading

Option 4: Using Docker

FROM python:3.13-rc

# Python RC images may include free-threading
# Check with: python -c "import sys; print(hasattr(sys, '_is_gil_enabled'))"

Verifying Free-Threading is Enabled

# Check if GIL is disabled
python3 -c "
import sys
if hasattr(sys, '_is_gil_enabled'):
    print('Free-threading supported:', not sys._is_gil_enabled())
else:
    print('Free-threading not available (Python < 3.13 or GIL enabled)')
"

Expected output for free-threaded Python:

Free-threading supported: True

Building erlang_python for Free-Threading

# Clean previous builds
rebar3 clean

# Set Python config path (adjust for your installation)
export PYTHON_CONFIG=$HOME/python-nogil/bin/python3-config

# Or if using pyenv
export PYTHON_CONFIG=$(pyenv prefix 3.13.0)/bin/python3-config

# Build
rebar3 compile

Verifying Free-Threading Mode

1> application:ensure_all_started(erlang_python).
{ok, [erlang_python]}

2> py:execution_mode().
free_threaded  % Should show 'free_threaded' instead of 'subinterp' or 'multi_executor'

3> py:num_executors().
1  % In free_threaded mode, no executor pool is used

Running Tests

# Run the test suite
rebar3 ct --suite=py_SUITE

# Run benchmarks
escript examples/benchmark.erl --full

Performance Comparison

Run benchmarks with both standard and free-threaded Python to compare:

Standard Python (with GIL)

# Use standard Python
export PYTHON_CONFIG=/usr/bin/python3-config
rebar3 clean && rebar3 compile
escript examples/benchmark.erl --concurrent

Free-Threaded Python

# Use free-threaded Python
export PYTHON_CONFIG=$HOME/python-nogil/bin/python3-config
rebar3 clean && rebar3 compile
escript examples/benchmark.erl --concurrent

Expected results:

  • Free-threaded mode should show higher throughput for concurrent CPU-bound workloads
  • The difference is most noticeable with many concurrent processes making Python calls

Caveats

Extension Compatibility

Not all Python C extensions are compatible with free-threading. Extensions that rely on the GIL for thread safety may crash or produce incorrect results.

Known compatible:

  • Standard library modules
  • NumPy (recent versions)

May have issues:

  • Older C extensions
  • Extensions using non-thread-safe C libraries

Memory Model

Free-threaded Python uses a different memory model. Be aware of:

  • Increased memory usage (per-object locks)
  • Different garbage collection behavior
  • Potential for data races in Python code

Testing Recommendations

  1. Start with unit tests: Ensure basic functionality works
  2. Test concurrency: Run concurrent benchmarks
  3. Check for crashes: Monitor for segfaults during heavy load
  4. Profile memory: Watch for memory leaks or bloat

Troubleshooting

Build Fails

error: Python.h not found

Ensure PYTHON_CONFIG points to the free-threaded Python installation:

ls $(dirname $(which python3))/../include/*/Python.h

Mode Shows 'multi_executor' Instead of 'free_threaded'

The Python build may not have Py_GIL_DISABLED defined. Verify:

python3 -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"

Should print 1 for free-threaded builds.

Crashes Under Load

Some extensions may not be thread-safe. Try:

  1. Isolate the problematic extension
  2. Check if a thread-safe version exists
  3. Fall back to sub-interpreter mode for those calls

See Also