View Source Image.Classification (image v0.39.0)

Implements image classification functions using Axon machine learning models managed by Bumblebee.

Image classification refers to the task of extracting information from an image. In this module, the information extracted is one or more labels desribing the image. Typically something like "sports car" or "Blenheim spaniel". The labels returns depend on the machine learning model used.

Configuration

The machine learning model to be used is configurable. The :model and :featurizer may be any model supported by Bumblebee. The :name is the name given to the classification service process.

The default configuration is:

# runtime.exs
config :image, :classifier,
  model: {:hf, "microsoft/resnet-50"},
  featurizer:  {:hf, "microsoft/resnet-50"},
  name: Image.Classification.Server,
  autostart: true

Autostart

If autostart: true is configured (the default) then a process is started under a supervisor to execute the classification requests. If running the process under an application supervision tree is desired, set autostart: false. In that case the function classifer/1 can be used to return a Supervisor.child_spec/0.

Adding a classification server to an application supervision tree

To add image classification to an application supervision tree, use Image.Classification.classifier/1 to return a child spec: For example:

# Application.ex
def start(_type, _args) do
  children = [
    # default classifier configuration
    Image.Classification.classifier()

    # custom classifier configuration
    Image.Classification.classifier(model: {:hf, "google/vit-base-patch16-224"},
      featurizer: {:hf, "google/vit-base-patch16-224"})
  ]

  Supervisor.start_link(
    children,
    strategy: :one_for_one
  )
end

Note

This module is only available if the optional dependency Bumblebee is configured in mix.exs.

Summary

Functions

classifier(classifier \\ Application.get_env(:image, :classifier, []))

Returns a child spec suitable for starting an image classification process as part of a supervision tree.

classify(image, options \\ [])

Classify an image using a machine learning model.

labels(image, options \\ [])

Classify an image using a machine learning model and return the labels that meet a minimum score.

Functions

Link to this function

classifier(classifier \\ Application.get_env(:image, :classifier, []))

View Source 
@spec classifier(configuration :: Keyword.t()) :: {Nx.Serving, Keyword.t()}

Returns a child spec suitable for starting an image classification process as part of a supervision tree.

Arguments

  • configuration is a keyword list.The default is Application.get_env(:image, :classifier, []).

Configuration keys

  • :model is any supported machine learning model for image classification supported by Bumblebee.

  • :featurizer is any supported machine learning model for image featurization supported by Bumblebee.

  • :name is the name given to the classification process when it is started.

Default configuration

The default configuration is:

[
  model: {:hf, "microsoft/resnet-50"},
  featurizer: {:hf, "microsoft/resnet-50"},
  name: Image.Classification.Server
]
Link to this function

classify(image, options \\ [])

View Source (since 0.18.0) 
@spec classify(image :: Vix.Vips.Image.t(), Keyword.t()) ::
  %{predictions: [%{label: String.t(), score: float()}]}
  | {:error, Image.error_message()}

Classify an image using a machine learning model.

Arguments

Options

  • :backend is any valid Nx backend. The default is Nx.default_backend/0.

  • :server is the name of the process performing the classification service. The default is Image.Classification.Server.

Returns

  • A map containing the predictions of the image classification.

Example 

iex> puppy = Image.open!("./test/support/images/puppy.webp")
iex> %{predictions: [%{label: "Blenheim spaniel", score: _} | _rest]} =
...>   Image.Classification.classify(puppy)
Link to this function

labels(image, options \\ [])

View Source (since 0.18.0) 
@spec labels(image :: Vix.Vips.Image.t(), options :: Keyword.t()) ::
  [String.t()] | {:error, Image.error_message()}

Classify an image using a machine learning model and return the labels that meet a minimum score.

Arguments

Options

  • :backend is any valid Nx backend. The default is Nx.default_backend/0.

  • :min_score is the minimum score, a float between 0 and 1, which a label must match in order to be returned.

Returns

  • A list of labels. The list may be empty if there are no predictions that exceed the :min_score.

  • {:error, reason}

Example 

iex> {:ok, image} = Image.open ("./test/support/images/lamborghini-forsennato-concept.jpg")
iex> Image.Classification.labels(image)
["sports car", "sport car"]