Contributing to Beaver

This document describes how to contribute to Beaver by introducing the idea behind how Beaver group its functionalities and how to set up the development environment.

Beaver's functionalities

There are three main parts in Beaver. Here is a brief introduction to each part:

  • DSL: The syntax of Beaver SSA expression.
  • Utilities: Work with MLIR in Elixir way.
  • Bindings: Thin wrapper of MLIR CAPI.

DSL

Modules including Beaver, Beaver.Env.

DSL is the core part of Beaver. It uses Elixir's syntax to express MLIR semantics.

Utilities

Modules including Beaver.Walker, Beaver.Composer

Utilities are the helper functions that help to generate or manipulate MLIR IR. They are implemented in Elixir and is designed to be used in the DSL part to further enhance it and improve ergonomics.

Bindings

Modules including Beaver.MLIR, Beaver.MLIR.Dialect, Beaver.MLIR.Pass, Beaver.MLIR.Transform, Beaver.MLIR.ExecutionEngine

Bindings are the part that provides the interface to the MLIR CAPIs. It is implemented in Zig and is responsible for calling MLIR functions. Note that Beaver's bindings will try not to use TableGen and instead try to make use Elixir and Zig's meta-programming features to generate the bindings.

Development

  1. Install Elixir, see installation guide
  2. Install Zig, see installation guide
  3. Install LLVM/MLIR

  • Option 1: Install with pip

    python3 -m pip install -r dev-requirements.txt
export LLVM_CONFIG_PATH=$(python3 -c 'import mlir;print(mlir.__path__[0])')/bin/llvm-config

  • Option 2: Build from source https://mlir.llvm.org/getting_started/ Recommended install commands:

    cmake -B build -S llvm -G Ninja -DLLVM_ENABLE_PROJECTS=mlir \
  -DLLVM_TARGETS_TO_BUILD="host" \
  -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_ENABLE_OCAMLDOC=OFF \
  -DLLVM_ENABLE_BINDINGS=OFF \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_INSTALL_PREFIX=${HOME}/llvm-install
cmake --build build -t install
export LLVM_CONFIG_PATH=$HOME/llvm-install/bin/llvm-config

    (Optional) To use Vulkan:

    • Install Vulkan SDK (global installation is required), reference: https://vulkan.lunarg.com/sdk/home

    • Setting environment variable by adding commands these to your bash/zsh profile:

      # you might need to change the version here
cd $HOME/VulkanSDK/1.3.216.0/
source setup-env.sh
cd -

    • Use vulkaninfo and vkvia to verify Vulkan is working

    • Add -DMLIR_ENABLE_VULKAN_RUNNER=ON in LLVM CMake config command

  1. Develop and run tests

  • Clone this repo and kinda in the same directory

    git clone https://github.com/beaver-lodge/beaver.git
git clone https://github.com/beaver-lodge/kinda.git

  • Make sure LLVM environment variable is set properly, as otherwise it might fail to build

    echo $LLVM_CONFIG_PATH

  • Build and run Elixir tests

    mix deps.get
BEAVER_BUILD_CMAKE=1 mix test
# run tests with filters
mix test --exclude vulkan # use this to skip vulkan tests
mix test --only smoke
mix test --only nx
  1. debug
  • setting environment variable to control Erlang scheduler number, ERL_AFLAGS="+S 10:5"
  • run mix test under LLDB, scripts/lldb-mix-test

Release a new version

Update Elixir source

Linux

  • Run CI, which generates the new GitHub release uploaded to https://github.com/beaver-lodge/beaver-prebuilt/releases.
  • Update release url in mix.exs
  • Run docker image to build for ARM:
    docker run -it --rm -v $PWD/..:/src -w /src/beaver --env MIX_BUILD_ROOT='_build/arm' jackalcooper/beaver-livebook-arm64:latest bash scripts/build-for-publish.sh

Mac

  • Run macOS build with:

    rm -rf _build/prod
bash scripts/build-for-publish.sh

  • Upload the beaver-nif-[xxx].tar.gz file to release

Generate checksum.exs 

rm checksum.exs
mix clean
mix
mix elixir_make.checksum --all --ignore-unavailable --print

Check the version in the output is correct.

Publish to Hex 

BEAVER_BUILD_CMAKE=1 mix hex.publish

Format CMake files 

python3 -m pip install cmake-format
cmake-format -i native/**/CMakeLists.txt native/**/*.cmake