# Text classification — language identification

`Text.Language.Classifier.Fasttext` is a pure-Elixir port of fastText's [`lid.176`](https://fasttext.cc/docs/en/language-identification.html) model — a supervised classifier trained on Wikipedia, Tatoeba, and SETimes data that recognises **176 languages** from short input. The implementation is validated bit-for-bit against the official C++/Python reference for hashing, n-gram extraction, feature assembly, and tree traversal: same input, same prediction, same probabilities (within float-32 rounding).

It runs entirely in the BEAM — no NIFs, no Python sidecar, no model server. The trade-off is the model file (~126 MB) is fetched once at install time and lives on disk; this guide walks through the setup and the API.

## One-time setup

The `lid.176.bin` model file is **not** part of the Hex package — every install fetches its own copy. Run once after adding `:text` to your dependencies:

```sh
mix text.download_lid176
```

The file lands at `priv/lid_176/lid.176.bin` inside the project. It's gitignored and not committed.

For production environments that want every external artefact present at boot, use the broader `mix text.download_models` task — same fetch, but it can also pre-download the Bumblebee models used by `Text.Sentiment`, `Text.POS`, `Text.NER`, and `Text.WordCloud.Backends.KeyBERT`.

## Loading the model

The model is loaded once per VM and reused across every detection call:

```elixir
{:ok, model} =
  Text.Language.Classifier.Fasttext.ModelLoader.load(
    Path.join(:code.priv_dir(:text), "lid_176/lid.176.bin")
  )
```

The result is a `Text.Language.Classifier.Fasttext.Model` struct holding the input matrix, output matrix, dictionary, and Huffman tree (if applicable) as `Nx` tensors. Loading takes a few seconds; a typical pattern is to load at application boot and stash the model in `:persistent_term` or a `GenServer` for the rest of the VM's lifetime.

## Detecting a language

`detect/3` returns a full `Detection` struct:

```elixir
{:ok, det} = Text.Language.Classifier.Fasttext.detect("Bonjour le monde", model)

det.language     #=> "fr"
det.script       #=> :Latn
det.confidence   #=> 0.984
det.alternatives #=> [{"en", 0.0035}, {"it", 0.0024}, {"oc", 0.0009}, {"ca", 0.0006}]
det.text         #=> "Bonjour le monde"
```

The struct fields:

| Field | Meaning |
|---|---|
| `:language` | BCP-47 language subtag (`"fr"`, `"zh"`, `"sr"`). |
| `:confidence` | Probability of the top prediction in `[0.0, 1.0]`. |
| `:script` | Unicode script atom derived from the input text (`:Latn`, `:Cyrl`, `:Hans`, `:Hant`, `:Hani`, …). Used downstream to disambiguate multi-script locales. |
| `:alternatives` | List of `{language, probability}` for the next-best predictions. |
| `:text` | The original input, preserved for downstream use. |

Common options:

* `:k` — number of top predictions to return. Default `5`. The first becomes the main `:language`; the rest fill `:alternatives`.

* `:threshold` — drop predictions below this probability. Default `0.0`. Raise it (e.g. `0.5`) to get `{:error, :no_predictions}` for ambiguous inputs you'd rather skip than guess at.

```elixir
case Text.Language.Classifier.Fasttext.detect(unknown_text, model, threshold: 0.5) do
  {:ok, det} -> route_by_language(det.language)
  {:error, :no_predictions} -> ask_user_to_clarify()
end
```

## Just the language code

When you only need the answer:

```elixir
{:ok, "es"} = Text.Language.Classifier.Fasttext.classify("Hola, ¿cómo estás?", model)
{:ok, "ru"} = Text.Language.Classifier.Fasttext.classify("Привет, мир!", model)
{:ok, "ja"} = Text.Language.Classifier.Fasttext.classify("こんにちは世界", model)
```

`classify/2` is a thin wrapper around `detect(text, model, k: 1)` that drops everything except the top language code. Useful for routing logic where you only care which bucket to send the text into.

## Resolving to a CLDR locale

`detect/3` returns a bare language code; downstream localisation systems usually want a full locale string like `zh-Hans-CN` or `fr-FR`. `to_locale/2` runs the detection through CLDR's likely-subtags algorithm to fill in the missing pieces:

```elixir
{:ok, det} = Text.Language.Classifier.Fasttext.detect("你好世界，这是简体中文。", model)
{:ok, "zh-Hans-CN"} = Text.Language.Classifier.Fasttext.to_locale(det)

{:ok, det} = Text.Language.Classifier.Fasttext.detect("你好世界，這是繁體中文。", model)
{:ok, "zh-Hant-TW"} = Text.Language.Classifier.Fasttext.to_locale(det)
```

When the optional [`localize`](https://hex.pm/packages/localize) dependency is loaded, this calls into CLDR's actual likely-subtags table. Without it, a built-in fallback table covers ~60 of the most common languages. Add `:localize` for production-grade locale resolution:

```elixir
{:localize, "~> 0.23", optional: true}
```

Override the inferred region or script:

```elixir
{:ok, "fr-Latn-CA"} = Text.Language.Classifier.Fasttext.to_locale(det, region: :CA)
```

The region option is typically wired to an `Accept-Language` header or IP geolocation when available; otherwise the CLDR default for the language wins.

## Script detection and Hans/Hant

Many languages are written in more than one script (Serbian in Latin or Cyrillic, Punjabi in Gurmukhi or Shahmukhi, Chinese in Simplified or Traditional Han). The fastText model returns a bare language code like `"zh"` — it doesn't distinguish `Hans` from `Hant`. `Text.Language.Classifier.Fasttext.ScriptDetector` runs alongside `detect/3` and contributes the script signal.

For Chinese specifically, `ScriptDetector` runs a second-pass codepoint-frequency analysis against curated lists of distinguishing characters. If the input contains characters present only in Simplified (`国`, `电`, `时`) it returns `:Hans`; if it contains Traditional-only characters (`國`, `電`, `時`) it returns `:Hant`. Inputs containing only shared Han codepoints fall back to `:Hani`, and likely-subtags then resolves to `Hans-CN` (the mainland-China default).

```elixir
{:ok, det} = Text.Language.Classifier.Fasttext.detect("国家电网", model)
det.script  #=> :Hans

{:ok, det} = Text.Language.Classifier.Fasttext.detect("國家電網", model)
det.script  #=> :Hant

{:ok, det} = Text.Language.Classifier.Fasttext.detect("人之初", model)
det.script  #=> :Hani  (shared codepoints — could be either)
```

## Confidence calibration

fastText's confidence scores are well-calibrated for *long* inputs (a sentence or more) but inflate aggressively on very short inputs. Common patterns:

* **Short noun phrases** ("Hello world") often produce confidence > 0.95 — usually correct, but sometimes overconfident on names that look multilingual.

* **Mixed-language text** ("Click the button to login") usually classifies as the dominant language with moderate confidence; check `:alternatives` if the result looks suspicious.

* **Code-mixed or transliterated text** ("kaisi ho?" written in Latin script for Hindi) often classifies as the script's default language (`:en`) rather than the intended one. Consider a higher `:threshold` and a fallback path for ambiguous cases.

For robust routing, look at the gap between top-1 and top-2 confidences in `:alternatives`. A small gap (< 0.1) signals genuine ambiguity even when the top score is high.

## Performance

The model's input matrix is ~128 MB of `float32` data held in an `Nx` tensor. The inference forward pass (`take + mean + dot`, plus the softmax tail for softmax-loss models) is wrapped in `Nx.Defn` so an EXLA-compiled execution runs the whole pass as a single fused XLA kernel.

**Per-prediction wall time on `lid.176`:**

| Backend | Time |
|---|---|
| `Nx.BinaryBackend` (no `:exla`) | ~600 µs |
| `EXLA.Backend`, no defn fusion | ~200 µs |
| `EXLA.Backend` + fused defn graph (default) | **~100 µs** |

For production throughput add `:exla` to your deps and configure it as both the default backend and the default `defn` compiler:

```elixir
# config/config.exs
config :nx, default_backend: EXLA.Backend
config :nx, :default_defn_options, compiler: EXLA
```

Without EXLA the package still works correctly — `Nx.Defn.Evaluator` runs the same `defn` graph against `Nx.BinaryBackend` — but per-prediction wall time is roughly an order of magnitude higher.

## The 176 supported languages

The full set is documented at the [fastText project page](https://fasttext.cc/docs/en/language-identification.html). Coverage includes all 24 official EU languages, every UN official language, the major South and Southeast Asian languages, and a long tail of regional and minority languages. Languages **not** in `lid.176` include very recently-added minority languages (Quechua, some indigenous American languages) and constructed languages outside Esperanto and Ido.

Use `model.dictionary.nlabels` (which equals 176) and `model.labels` (a list of every supported label) to enumerate at runtime if you need a UI selector or to validate a user's expected language.

## Putting it together

A typical production wiring:

```elixir
# At app boot, in your Application.start/2:
def start(_type, _args) do
  {:ok, model} = Text.Language.Classifier.Fasttext.ModelLoader.load(
    Path.join(:code.priv_dir(:text), "lid_176/lid.176.bin")
  )
  :persistent_term.put(MyApp.LidModel, model)
  Supervisor.start_link(children, opts)
end

# At call site:
def detect_language(text) do
  model = :persistent_term.get(MyApp.LidModel)

  case Text.Language.Classifier.Fasttext.detect(text, model, threshold: 0.5) do
    {:ok, det} ->
      {:ok, locale} = Text.Language.Classifier.Fasttext.to_locale(det)
      {:ok, det.language, locale}

    {:error, :no_predictions} ->
      {:error, :ambiguous}
  end
end
```

This pattern loads the model once, keeps it warm in `:persistent_term`, and produces fully-resolved CLDR locales on every call.