Settings View Source cimg_ex

A lightweight image processing module in Elixir using CImg, aimed at creating auxiliary routines for Deep Learning.

Note: Only a few image processing functions are available yet.

Design detail

Image data handled by CImg is placed on the NIF side. On the Elixir side, image processing is performed through RESOURCE of images - generated by NIF - stored in the %CImg{} structure. You cannot directly read or process pixels in an image. Instead, you must use the image processing functions provided by the CImg module.

Images are automatically subject to garbage collection when they are no longer used. The NIFs RESOURCE management mechanism is used to achieve this functionality.

Each function provided by CImg can be used alone or as a set of functions. In the latter case, a seed image is first prepared using CImg.builder/1 or similar, and then a series of functions are put together using Elixir's pipe syntax.

Platform

I have confirmed it works in the following OS environment.

• Windows MSYS2/MinGW64
• WSL2/Ubuntu 20.04, 18.04
• Linux Mint 20 "Ulyana"
• Nerves rpi

Requirements

python3 is required to build this module.

In addition, the following libraries are required to display images on a PC screen.

• GDI32 on Windows
• X11 on Linux

Installation

Add following dependency to your mix.exs.

def deps do
  [
    {:cimg, "~> 0.1.19"}
  ]
end

and install dependencies:

$ mix deps.get
$ mix deps.compile

Demo

A sample program is in the demo directory. You can run it by following the steps below.

$ cd demo
$ mix deps.get
$ mix run -e "CImgDemo.demo1"

Closing the application window will stop the demo program.

Another demo:

$ mix run -e "CImgDemo.demo2"

$ mix run -e "CImgDemo.demo3"

Example: Deep Labelling for Semantic Image Segmentation

The following code is an excerpt of the inference portion from the DeepLab app implemented using TflInterp. CImg preprocesses the input image and passes it through the DeepLab model. And then, the tensor output by DeepLab is transformed by CImg into a colormaedp image.

defmodule TflDemo.DeepLab3 do
  alias TflDemo.DeepLab3.Prediction

  def apply_deeplab3(img_file) do
    img = CImg.load(img_file)

    segments = Prediction.apply(img)
  end
end


defmodule TflDemo.DeepLab3.Prediction do
  use TflInterp, model: "priv/lite-model_deeplabv3_1_metadata_2.tflite"
  
  @deeplab3_shape {257, 257}

  def apply(img) do
    # preprocess
    bin = img
      |> CImg.resize(@deeplab3_shape)
      |> CImg.to_binary(range: {-1.0, 1.0})

    # prediction
    outputs =
      __MODULE__
      |> TflInterp.set_input_tensor(0, bin)
      |> TflInterp.invoke()
      |> TflInterp.get_output_tensor(0)
      |> Nx.from_binary({:f, 32}) |> Nx.reshape({257, 257, :auto})
      
    # postprocess
    outputs
    |> Nx.argmax(axis: 2)
    |> Nx.as_type({:u, 8})
    |> Nx.to_binary()
    |> CImg.from_binary(257, 257, 1, 1, "<u1")
    |> CImg.color_mapping(:lines)    # convert into a colormaped image
  end
end

License

cimg_ex is licensed under the Apache License Version 2.0.

-- license overview of included 3rd party libraries --

• The "CImg" Library is licensed under the CeCILL-C/CeCILL.
• The "stb" - single-file public domain libraries for C/C++ - is public domain or MIT licensed.