View Source TflInterp

Tensorflow lite interpreter for Elixir. Deep Learning inference framework for embedded devices.

Design policy (Features)

TflInterp is designed based on the following policy.

  1. Provide only the Deep Learning inference. It aims to the poor-resource devices such as IOT and mobile.
  2. Easy to understand. The inference part, excluding pre/post-processing, can be written in a few lines.
  3. Use trained models from major Deep Learning frameworks that are easy to obtain.
  4. Multiple inference models can be used from a single application.
  5. There are few dependent modules. It does not have image processing or matrix calculation functions.
  6. TflInterp does not block the erlang/elixir process scheduler. It runs as an OS process outside of elixir.
  7. The back-end inference engine can be easily replaced. It's easy to keep up with the latest Deep Learninig technology.

And I'm trying to make TflInterp easy to install.

short or concise history

The development of Tflinterp started in 2020 Nov. The original idea was to use Nerves to create an AI remote controlled car. In the first version, I implemented Yolo3, but the design strongly depended on that model, which made it difficult to use in other applications. Reflecting on that mistake, I redesigned Tflinterp according to the above design guidelines.

Platform

It has been confirmed to work in the following OS environment.

  • Windows
  • WSL2/Ubuntu 20.04
  • Nerves ARMv6, ARMv7NEON and AArch64

Requirements

  • cmake 3.21.0 or later
  • git
  • Visual C++ 2022(Windows)

Installation

Since 0.1.3, the installation method of this module has changed. You may need to remove previously installed TflInterp before installing new version.

There are two installation methods. You can choose either one according to your purpose.

Method-1. Like any other elixir module, add TflInterp to the dependency list in the mix.exs.

def deps do
  [
    ...
    {:tfl_interp, "~> 0.1.16"},
  ]
end

Method-2. Download TflInterp to a directory and build it ahead of time.

# download TflInterp in advance.
$ cd /home/{your home}/workdir
$ git clone -b nerves https://github.com/shoz-f/tfl_interp.git
$ cd tfl_interp
$ mix deps.get
$ mix compile

After adding the directory path to the dependency list in your mix.exs file, add the following line to the deps() function: System.put_env("NNCOMPILED", "YES")

This line sets an environment variable that instructs your TflInterp application to use the precompiled tfl_interp.exe file, eliminating the need to build it again.

def deps do
  System.put_env("NNCOMPILED", "YES)
  [
    ...
    {:tfl_interp, path: "/home/{your home}/workdir/tfl_interp"}
  ]
end

Then you run the following commands in your application project.

For Desktop application:

$ mix deps.get
$ mix compile

For Nerves application:

$ export MIX_TARGET=rpi3  # <- specify target device tag
$ mix deps.get
$ mix firmware

It takes a long time to finish the build. Because it will download the required files - Tensorflow sources, ARM toolchain [^1], etc - at the first build time. Method 1 saves the downloaded files under "{your app}/deps/tfl_interp". On the other hand, method 2 saves them under "/home/{your home}/workdir/tfl_interp". If you want to reuse the downloaded files in other applications, I recommend Method 2.

In either method 1 or 2, the external modules required for Tensorflow lite are stored under their "_build/{target}/.cmake_build" according to the cmakelists.txt that comes with Tensorflow.

[^1] Unfortunately, the ARM toolchain that comes with Nerves can not build Tensorflow lite. We need to get the toolchain recommended by the Tensorflow project.

After installation, you will have the directory tree like these:

Method 1

work_dir
  +- your-app
       +- _build/
       |    +- dev/
       |         +- .cmake_build/ --- CMakeCash.txt and external modules that Tensorflowlite depends on.
       |         |                    The cmake build outputs are stored here also.
       |         +- lib/
       |         |    +- tfl_interp
       |         |         +- ebin/
       |         |         +- priv
       |         |              +- tfl_interp --- executable: tensorflow interpreter.
       |         :
       |
       +- deps/
       |    + tfl_interp
       |    |   +- 3rd_party/ --- Tensorflow sources, etc.
       |    |   +- lib/ --- TflInterp module.
       |    |   +- src/ --- tfl_interp C++ sources.
       |    |   +- test/
       |    |   +- toolchain/ --- ARM toolchains for Nerves.
       |    |   +- CMakeLists.txt --- CMake configuration for for building tfl_interp.
       |    |   +- mix.exs
       |    :
       |
       +- lib/
       +- test/
       +- mix.exs

Method 2

work_dir
  +- your-app
  |    +- _build/
  |    |    +- dev/
  |    |         +- lib/
  |    |         |    +- tfl_interp
  |    |         |         +- ebin/
  |    |         |         +- priv
  |    |         |              +- tfl_interp --- executable: tensorflow interpreter.
  |    |         |                                copy it from the tfl_interp project.
  |    |         :
  |    |
  |    +- deps/
  |    +- lib/
  |    +- test/
  |    +- mix.exs
  |
  +- tfl_interp
       +- 3rd_party/ --- Tensorflow sources, etc.
       +- _build/
       |    +- dev/
       |         +- .cmake_build/ --- CMakeCash.txt and external modules that Tensorflowlite depends on.
       |         |                    The cmake build outputs are stored here also.
       |         +- lib/
       |         |    +- tfl_interp
       |         |         +- ebin/
       |         |         +- priv
       |         |              +- tfl_interp --- executable: tensorflow interpreter.
       |         :
       |
       +- deps/
       +- lib/ --- TflInterp module.
       +- src/ --- tfl_interp C++ sources.
       +- test/
       +- toolchain/ --- ARM toolchains for Nerves.
       +- CMakeLists.txt --- CMake configuration for for building tfl_interp.
       +- mix.exs

Basic Usage

You get the trained tflite model and save it in a directory that your application can read. "your-app/priv" may be good choice.

$ cp your-trained-model.tflite ./priv

Next, you will create a module that interfaces with the deep learning model. The module will need pre-processing and post-processing in addition to inference processing, as in the example following. TflInterp provides inference processing only.

You put use TflInterp at the beginning of your module, specify the model path as an optional argument. In the inference section, you will put data input to the model (TflInterp.set_input_tensor/3), inference execution (TflInterp.invoke/1), and inference result retrieval (TflInterp.get_output_tensor/2).

defmodule YourApp.YourModel do
  use TflInterp, model: "priv/your-trained-model.tflite"

  def predict(data) do
    # preprocess
    #  to convert the data to be inferred to the input format of the model.
    input_bin = convert-float32-binaries(data)

    # inference
    #  typical I/O data for Tensorflow lite models is a serialized 32-bit float tensor.
    output_bin =
      __MODULE__
      |> TflInterp.set_input_tensor(0, input_bin)
      |> TflInterp.invoke()
      |> TflInterp.get_output_tensor(0)

    # postprocess
    #  add your post-processing here.
    #  you may need to reshape output_bin to tensor at first.
    tensor = output_bin
      |> Nx.from_binary({:f, 32})
      |> Nx.reshape({size-x, size-y, :auto})

    * your-postprocessing *
    ...
  end
end

Let's enjoy ;-)

Demo

There is MNIST web application in demo_mnist directory. You can do it by following the steps below.

$ cd demo_mnist
$ mix deps.get
$ mix run --no-halt

And then, please open your browser with "http://localhost:5000".

License

TflInterp is licensed under the Apache License Version 2.0.