Providers

LlmComposer ships with five built-in providers. Each implements the LlmComposer.Provider behaviour and can be used interchangeably.

Provider Overview

Feature Compatibility

¹ Ollama via the native provider does not support function calls or structured outputs. Use LlmComposer.Providers.OpenAI pointed at your Ollama instance's OpenAI-compatible endpoint (/v1/chat/completions) to access these features on supported models.

Common Options

All providers accept these options:

OpenAI

Application.put_env(:llm_composer, :open_ai, api_key: "<your api key>")

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenAI, [model: "gpt-4o"]}
  ],
  system_prompt: "You are a helpful assistant."
}

OpenAI Responses API

LlmComposer.Providers.OpenAIResponses targets the /responses endpoint and supports reasoning models, structured outputs via response_schema, and streaming:

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenAIResponses,
     [
       model: "gpt-5-nano",
       reasoning: %{effort: "low"}
     ]}
  ],
  system_prompt: "You are a concise assistant."
}

{:ok, res} = LlmComposer.simple_chat(settings, "Explain quantum computing in one paragraph")

Streaming with OpenAI Responses API:

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenAIResponses, [model: "gpt-4o-mini"]}
  ],
  system_prompt: "You are a helpful assistant.",
  stream_response: true
}

{:ok, res} = LlmComposer.simple_chat(settings, "Write one sentence about stars")

res.stream
|> LlmComposer.parse_stream_response(res.provider)
|> Enum.each(fn chunk -> IO.write(chunk.text || "") end)

OpenAI-Compatible Servers

vLLM, LocalAI, LM Studio, and Ollama's OpenAI-compatible endpoint can all be used by overriding :url:

Application.put_env(:llm_composer, :open_ai, url: "http://localhost:8000/v1", api_key: "token")

# or per-request:
provider_opts: [
  model: "mistral-7b",
  api_key: "ignored",
  url: "http://localhost:8000/v1"
]

For Ollama specifically:

Application.put_env(:llm_composer, :open_ai, url: "http://localhost:11434/v1", api_key: "ollama")

{LlmComposer.Providers.OpenAI, [model: "llama3.1"]}
# or
{LlmComposer.Providers.OpenAIResponses, [model: "llama3.1"]}

OpenRouter

OpenRouter gives access to many models through a single OpenAI-compatible API, with unique features like fallback models and provider routing.

Application.put_env(:llm_composer, :open_router, api_key: "<your openrouter api key>")

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenRouter,
     [
       model: "anthropic/claude-3-sonnet",
       models: ["openai/gpt-4.1", "fallback-model2"],
       provider_routing: %{order: ["openai", "azure"]}
     ]}
  ],
  system_prompt: "You are a helpful assistant."
}

Custom Headers

OpenRouter recommends sending HTTP-Referer and X-Title headers for rankings:

provider_opts: [
  model: "anthropic/claude-3-haiku",
  headers: [
    {"HTTP-Referer", "https://my-app.com"},
    {"X-Title", "My App"}
  ]
]

Ollama

No API key required. Start the Ollama server, then:

# Application.put_env(:llm_composer, :ollama, url: "http://localhost:11434")

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.Ollama, [model: "llama3.1"]}
  ],
  system_prompt: "You are a helpful assistant."
}

{:ok, res} = LlmComposer.simple_chat(settings, "hi")
IO.inspect(res.main_response)

Note: Ollama does not report token usage. input_tokens and output_tokens will be empty. Function calls and structured outputs are not available through the native Ollama provider — use the OpenAI-compatible endpoint approach instead (see OpenAI section above).

AWS Bedrock

Requires the optional {:ex_aws, "~> 2.6"} dependency. Credentials are read from ExAws config. See the ExAws documentation for all supported credential sources.

# config/config.exs
config :ex_aws,
  access_key_id: "your key",
  secret_access_key: "your secret"

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.Bedrock, [model: "eu.amazon.nova-lite-v1:0"]}
  ],
  system_prompt: "You are a helpful assistant."
}

{:ok, res} = LlmComposer.simple_chat(settings, "What is quantum computing?")
IO.inspect(res.main_response)

Per-Service Credentials

If you need different AWS credentials for Bedrock (e.g. a dedicated IAM user), scope them under the "bedrock-runtime" service key — they take precedence over the global config:

config :ex_aws,
  access_key_id: "GLOBAL_KEY",
  secret_access_key: "GLOBAL_SECRET"

config :ex_aws,
  "bedrock-runtime": [
    access_key_id: "BEDROCK_KEY",
    secret_access_key: "BEDROCK_SECRET",
    region: "eu-west-1"
  ]

Google

Google AI Studio (API Key)

Application.put_env(:llm_composer, :google, api_key: "<your google api key>")

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.Google, [model: "gemini-2.5-flash"]}
  ],
  system_prompt: "You are a helpful assistant."
}

{:ok, res} = LlmComposer.simple_chat(settings, "What is quantum computing?")
IO.inspect(res.main_response)

Vertex AI

Vertex AI requires OAuth 2.0 authentication via the Goth library.

Dependencies:

{:goth, "~> 1.4"}

Service Account Setup:

1. Create a service account in Google Cloud Console.
2. Grant it the Vertex AI User or Vertex AI Service Agent IAM role.
3. Download the JSON credentials file.

Basic Example:

google_json = File.read!(Path.expand("~/path/to/service-account.json"))
credentials = Jason.decode!(google_json)

{:ok, _pid} =
  Goth.start_link(
    source: {:service_account, credentials},
    name: MyApp.Goth
  )

Application.put_env(:llm_composer, :google, goth: MyApp.Goth)

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.Google,
     [
       model: "gemini-2.5-flash",
       vertex: %{
         project_id: "my-gcp-project",
         location_id: "us-central1"
       }
     ]}
  ],
  system_prompt: "You are a helpful assistant."
}

Production Setup (Supervision Tree):

# application.ex
defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    google_json = File.read!(Application.get_env(:my_app, :google_credentials_path))
    credentials = Jason.decode!(google_json)

    children = [
      {Goth, name: MyApp.Goth, source: {:service_account, credentials}}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

# config/config.exs
config :llm_composer, :google, goth: MyApp.Goth
config :my_app, :google_credentials_path, "/path/to/service-account.json"

Vertex AI Options:

Structured Outputs

Pass response_schema in provider options to get responses conforming to a JSON schema. Supported by OpenAI, OpenRouter, Google, and Bedrock.

settings = %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenRouter,
     [
       model: "google/gemini-2.5-flash",
       response_schema: %{
         "type" => "object",
         "properties" => %{
           "answer" => %{"type" => "string"},
           "confidence" => %{"type" => "number"}
         },
         "required" => ["answer"]
       }
     ]}
  ]
}

Custom Request Parameters

Use request_params to merge extra parameters into the provider request body. Supported by all providers.

# OpenAI — pass reasoning_effort
provider_opts: [
  model: "gpt-5-mini",
  request_params: %{reasoning_effort: "low"}
]

# OpenRouter — pass temperature and max_tokens
provider_opts: [
  model: "anthropic/claude-3-haiku",
  request_params: %{temperature: 0.3, max_tokens: 500}
]