Ollama Server Setup

Copy Markdown View Source

This guide covers installing Ollama, running the local server, pulling models, and connecting to local or cloud endpoints from the Elixir client.

Install Ollama

macOS and Windows

Download the installer for your platform:

Linux (quick install)

curl -fsSL https://ollama.com/install.sh | sh

Linux (manual install)

If you are upgrading from an older version, remove the old libraries first:

sudo rm -rf /usr/lib/ollama

Download and extract the package:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tgz \
  | sudo tar zx -C /usr

Start the server:

ollama serve

Verify:

ollama -v

AMD GPU (ROCm)

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tgz \
  | sudo tar zx -C /usr

ARM64

curl -fsSL https://ollama.com/download/ollama-linux-arm64.tgz \
  | sudo tar zx -C /usr

Create a dedicated user and group:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

Create /etc/systemd/system/ollama.service:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=$PATH"

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl start ollama

Check status:

systemctl status ollama

GPU Drivers (Optional)

NVIDIA CUDA

Install CUDA drivers from:

Verify:

nvidia-smi

AMD ROCm

Install ROCm from:

For newer GPU support, consider the latest AMD drivers:

Customization

Edit the systemd service or add overrides:

sudo systemctl edit ollama

Example override (/etc/systemd/system/ollama.service.d/override.conf):

[Service]
Environment="OLLAMA_DEBUG=1"

Updating

Re-run the install script:

curl -fsSL https://ollama.com/install.sh | sh

Or re-download the package:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tgz \
  | sudo tar zx -C /usr

Installing a specific version

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh

Uninstall

Stop and remove the service:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service

Remove libraries and binaries:

sudo rm -r $(which ollama | tr 'bin' 'lib')
sudo rm $(which ollama)

Remove models and service user:

sudo userdel ollama
sudo groupdel ollama
sudo rm -r /usr/share/ollama

Start the Server

If you are not using systemd, start manually:

ollama serve

The default base URL is http://localhost:11434.

Pull a Model

ollama pull llama3.2

For examples, also pull:

ollama pull nomic-embed-text
ollama pull llava
ollama pull deepseek-r1:1.5b

Thinking examples use deepseek-r1:1.5b, which supports think.

Browse models:

List local models:

ollama list

Connect from Elixir

By default, the client targets http://localhost:11434:

client = Ollixir.init()

Override the host with OLLAMA_HOST:

export OLLAMA_HOST="http://ollama.internal:11434"

Or pass the URL directly:

client = Ollixir.init("http://ollama.internal:11434")

Cloud Models (via Local Ollama)

Cloud models let you run larger models while keeping the same local workflow. See https://ollama.com/search?c=cloud for the latest list.

  1. Sign in (one time):
ollama signin
  1. Pull a cloud model:
ollama pull gpt-oss:120b-cloud
  1. Use it like any other model:
{:ok, stream} = Ollixir.chat(client,
  model: "gpt-oss:120b-cloud",
  messages: [%{role: "user", content: "Why is the sky blue?"}],
  stream: true
)

stream
|> Stream.each(fn chunk ->
  IO.write(get_in(chunk, ["message", "content"]) || "")
end)
|> Stream.run()

Cloud API (ollama.com)

  1. Create an API key: https://ollama.com/settings/keys

  2. Export the key:

export OLLAMA_API_KEY="your_api_key_here"
  1. (Optional) List models:
curl https://ollama.com/api/tags
  1. Point the client at the hosted API:
client = Ollixir.init("https://ollama.com")

{:ok, response} = Ollixir.chat(client,
  model: "gpt-oss:120b",
  messages: [%{role: "user", content: "Why is the sky blue?"}]
)

Web search/fetch (requires API key)

web_search/2 and web_fetch/2 always call the hosted API and require OLLAMA_API_KEY. If the key is missing, these examples will skip; if the key is invalid, the API will return 401/403.

You can validate the key with:

curl -H "Authorization: Bearer $OLLAMA_API_KEY" https://ollama.com/api/tags

Cloud test runs (optional)

Cloud tests are tagged and excluded by default. After setting OLLAMA_API_KEY, you can run:

mix test --include cloud_api

Troubleshooting

  • Ollixir.ConnectionError: ensure ollama serve is running and reachable. The error message includes a download link if Ollama is not installed.
  • Wrong host/port: set OLLAMA_HOST or pass a URL to Ollixir.init/1.
  • Auth errors: confirm your API key and target host.

Next Steps