View Source OpenaiEx User Guide
Mix.install([
{:openai_ex, "~> 0.5.9"},
# {:openai_ex, path: Path.join(__DIR__, "..")},
{:kino, "~> 0.12.3"}
])
Introduction
OpenaiEx
is an Elixir library that provides a community-maintained OpenAI API client especially for Livebook development.
Portions of this project were developed with assistance from ChatGPT 3.5 and 4.
At this point, all API endpoints and features (as of Nov 15, 2023) are supported, including the Assistants API Beta, DALL-E-3, Text-to-Speech, the tools support in chat completions, and the streaming version of the chat completion endpoint. Streaming request cancellation is also supported.
Configuration of Finch pools and API base url are supported.
There are some differences compared to other elixir openai wrappers.
- I tried to faithfully mirror the naming/structure of the official python api. For example, content that is already in memory can be uploaded as part of a request, it doesn't have to be read from a file at a local path.
- I was developing for a livebook use-case, so I don't have any config, only environment variables.
- Streaming API versions, with request cancellation, are supported.
- The underlying transport is finch, rather than httpoison
- 3rd Party (including local) LLMs with an OpenAI proxy, as well as the Azure OpenAI API, are considered legitimate use cases.
To learn how to use OpenaiEx, you can refer to the relevant parts of the official OpenAI API reference documentation, which we link to throughout this document.
This file is an executable Livebook, which means you can interactively run and modify the code samples provided. We encourage you to open it in Livebook and try out the code for yourself!
Installation
You can install OpenaiEx using Mix:
In Livebook
Add the following code to the first connection cell:
Mix.install(
[
{:openai_ex, "~> 0.5.9"}
]
)
In a Mix Project
Add the following to your mix.exs file:
def deps do
[
{:openai_ex, "~> 0.5.9"}
]
end
Authentication
To authenticate with the OpenAI API, you will need an API key. We recommend storing your API key in an environment variable. Since we are using Livebook, we can store this and other environment variables as Livebook Hub Secrets.
apikey = System.fetch_env!("LB_OPENAI_API_KEY")
openai = OpenaiEx.new(apikey)
You can also specify an organization if you are a member of more than one:
# organization = System.fetch_env!("LB_OPENAI_ORGANIZATION")
# openai = OpenaiEx.new(apikey, organization)
For more information on authentication, see the OpenAI API Authentication reference.
Configuration
There are a few places where configuration seemed necessary.
Receive Timeout
The default receive timeout is 15 seconds. If you are seeing longer latencies, you can override the default with
# set receive timeout to 45 seconds
openai = OpenaiEx.new(apikey) |> OpenaiEx.with_receive_timeout(45_000)
Finch Instance
In production scenarios where you want to explicitly tweak the Finch pool, you can create a new Finch instance using
Finch.start_link(
name: MyConfguredFinch,
pools: ...
)
You can use this instance of Finch (instead of the default OpenaiEx.Finch
) by setting the finch name
openai_with_custom_finch = openai |> with_finch_name(MyConfiguredFinch)
Base Url
There are times, such as when using a local LLM (like Ollama) with an OpenAI proxy, when you need to reset the base url of the API. This is generally only applicable for chat and chat completion endpoints and can be accomplished by
# in this example, our development livebook server is running in a docker dev container while
# the local llm is running on the host machine
proxy_openai =
OpenaiEx.new(apikey) |> OpenaiEx.with_base_url("http://host.docker.internal:8000/v1")
Azure OpenAI
The Azure OpenAI API replicates the Completion, Chat Completion and Embeddings endpoints from OpenAI.
However, it modifies the base URL as well as the endpoint path, and adds a parameter to the URL query. These modifications are accommodated with the following calls:
for non Entra Id
openai = OpenaiEx._for_azure(azure_api_id, resource_name, deployment_id, api_version)
and for Entra Id
openai = OpenaiEx.new(entraId) |> OpenaiEx._for_azure(resource_name, deployment_id, api_version)
These methods will be supported as long as the Azure version does not deviate too far from the base OpenAI API.
Model
List Models
To list all available models, use the Model.list()
function:
alias OpenaiEx.Model
openai |> Model.list()
Retrieve Models
To retrieve information about a specific model, use the Model.retrieve()
function:
openai |> Model.retrieve("gpt-3.5-turbo")
For more information on using models, see the OpenAI API Models reference.
Chat Completion
To generate a chat completion, you need to define a chat completion request structure using the ChatCompletion.new()
function. This function takes several parameters, such as the model ID and a list of chat messages. We have a module ChatMessage
which helps create messages in the chat format.
alias OpenaiEx.ChatCompletion
alias OpenaiEx.ChatMessage
alias OpenaiEx.MsgContent
chat_req =
ChatCompletion.new(
model: "gpt-3.5-turbo",
messages: [
ChatMessage.user(
"Give me some background on the elixir language. Why was it created? What is it used for? What distinguishes it from other languages? How popular is it?"
)
]
)
You are able to pass images to the API by creating a message.
ChatMessage.user(
MsgContent.image_url(
"https://raw.githubusercontent.com/restlessronin/openai_ex/main/assets/images/starmask.png"
)
)
You can generate a chat completion using the ChatCompletion.create()
function:
chat_response = openai |> ChatCompletion.create(chat_req)
For a more in-depth example of ChatCompletion
, check out the Deeplearning.AI OrderBot Livebook.
You can also call the endpoint and have it stream the response. This returns the result as a series of tokens, which have to be put together in code.
To use the stream option, call the ChatCompletion.create()
function with stream: true
chat_stream = openai |> ChatCompletion.create(chat_req, stream: true)
IO.puts(inspect(chat_stream))
IO.puts(inspect(chat_stream.task_pid))
chat_stream.body_stream |> Stream.flat_map(& &1) |> Enum.each(fn x -> IO.puts(inspect(x)) end)
The chat_stream.task_pid
can be used in conjunction with OpenaiEx.HttpSse.cancel_request/1
to cancel an ongoing request.
For a detailed example of the use of the streaming ChatCompletion
API, including how to cancel an ongoing request, check out Streaming Orderbot, the streaming equivalent of the prior example.
For more information on generating chat completions, see the OpenAI API Chat Completions reference.
Function(Tool) Calling
In OpenAI's ChatCompletion
endpoint, you can use the function calling feature to call a custom function and pass its result as part of the conversation. Here's an example of how to use the function calling feature:
First, we set up the function specification and completion request. The function specification defines the name, description, and parameters of the function we want to call. In this example, we define a function called get_current_weather
that takes a location
parameter and an optional unit
parameter. The completion request includes the function specification, the conversation history, and the model we want to use.
tool_spec =
Jason.decode!("""
{"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"]
}
},
"required": ["location"]
}
}
}
""")
rev_msgs = [
ChatMessage.user("What's the weather like in Boston today?")
]
fn_req =
ChatCompletion.new(
model: "gpt-3.5-turbo",
messages: rev_msgs |> Enum.reverse(),
tools: [tool_spec],
tool_choice: "auto"
)
Next, we call the OpenAI endpoint to get a response that includes the function call.
fn_response = openai |> ChatCompletion.create(fn_req)
We extract the function call from the response and call the appropriate function with the given parameters. In this example, we define a map of functions that maps function names to their implementations. We then use the function name and arguments from the function call to look up the appropriate function and call it with the given parameters.
fn_message = fn_response["choices"] |> Enum.at(0) |> Map.get("message")
tool_call = fn_message |> Map.get("tool_calls") |> List.first()
tool_id = tool_call |> Map.get("id")
fn_call = tool_call |> Map.get("function")
functions = %{
"get_current_weather" => fn location, unit ->
%{
"location" => location,
"temperature" => "72",
"unit" => unit,
"forecast" => ["sunny", "windy"]
}
|> Jason.encode!()
end
}
fn_name = fn_call["name"]
fn_args = fn_call["arguments"] |> Jason.decode!()
location = fn_args["location"]
unit = unless is_nil(fn_args["unit"]), do: fn_args["unit"], else: "fahrenheit"
fn_value = functions[fn_name].(location, unit)
We then pass the returned value back to the ChatCompletion endpoint with the conversation history to that point to get the final response.
latest_msgs = [ChatMessage.tool(tool_id, fn_name, fn_value) | [fn_message | rev_msgs]]
fn_req_2 =
ChatCompletion.new(
model: "gpt-3.5-turbo",
messages: latest_msgs |> Enum.reverse()
)
fn_response_2 = openai |> ChatCompletion.create(fn_req_2)
The final response includes the result of the function call integrated into the conversation.
Image
Generate Image
We define the image creation request structure using the Image.new
function
alias OpenaiEx.Image
img_req = Image.Generate.new(prompt: "An adorable baby sea otter", size: "256x256", n: 1)
Then call the Image.create()
function to generate the images.
img_response = openai |> Image.generate(img_req)
For more information on generating images, see the OpenAI API Image reference.
Fetch the generated images
With the information in the image response, we can fetch the images from their URLs
fetch_blob = fn url ->
Finch.build(:get, url) |> Finch.request!(OpenaiEx.Finch) |> Map.get(:body)
end
fetched_images = img_response["data"] |> Enum.map(fn i -> i["url"] |> fetch_blob.() end)
View the generated images
Finally, we can render the images using Kino
fetched_images
|> Enum.map(fn r -> r |> Kino.Image.new("image/png") |> Kino.render() end)
img_to_expmt = fetched_images |> List.first()
Edit Image
We define an image edit request structure using the Image.Edit.new()
function. This function requires an image and a mask. For the image, we will use the one that we received. Let's load the mask from a URL.
# if you're having problems downloading raw github content, you may need to manually set your DNS server to "8.8.8.8" (google)
star_mask =
fetch_blob.(
"https://raw.githubusercontent.com/restlessronin/openai_ex/main/assets/images/starmask.png"
)
# star_mask = OpenaiEx.new_file(path: Path.join(__DIR__, "../assets/images/starmask.png"))
Set up the image edit request with image, mask and prompt.
img_edit_req =
Image.Edit.new(
image: img_to_expmt,
mask: star_mask,
size: "256x256",
prompt: "Image shows a smiling Otter"
)
We then call the Image.create_edit()
function
img_edit_response = openai |> Image.edit(img_edit_req)
and view the result
img_edit_response["data"]
|> Enum.map(fn i -> i["url"] |> fetch_blob.() |> Kino.Image.new("image/png") |> Kino.render() end)
Image Variations
We define an image variation request structure using the Image.Variation.new()
function. This function requires an image.
img_var_req = Image.Variation.new(image: img_to_expmt, size: "256x256")
Then call the Image.create_variation()
function to generate the images.
###
img_var_response = openai |> Image.create_variation(img_var_req)
img_var_response["data"]
|> Enum.map(fn i -> i["url"] |> fetch_blob.() |> Kino.Image.new("image/png") |> Kino.render() end)
For more information on images variations, see the OpenAI API Image Variations reference.
Embedding
Define the embedding request structure using Embedding.new
.
alias OpenaiEx.Embedding
emb_req =
Embedding.new(
model: "text-embedding-ada-002",
input: "The food was delicious and the waiter..."
)
Then call the Embedding.create()
function.
emb_response = openai |> Embedding.create(emb_req)
For more information on generating embeddings, see the OpenAI API Embedding reference
Audio
alias OpenaiEx.Audio
Create speech
For text to speech, we create an Audio.Speech
request structure as follows
speech_req =
Audio.Speech.new(
model: "tts-1",
voice: "alloy",
input: "The quick brown fox jumped over the lazy dog",
response_format: "mp3"
)
We then call the Audio.Speech.create()
function to create the audio response
speech_response = openai |> Audio.Speech.create(speech_req)
We can play the response using the Kino
Audio widget.
speech_response |> Kino.Audio.new(:mp3)
Create transcription
To define an audio transcription request structure, we need to create a file parameter using Audio.File.new()
.
# if you're having problems downloading raw github content, you may need to manually set your DNS server to "8.8.8.8" (google)
audio_url = "https://raw.githubusercontent.com/restlessronin/openai_ex/main/assets/transcribe.mp3"
audio_file = OpenaiEx.new_file(name: audio_url, content: fetch_blob.(audio_url))
# audio_file = OpenaiEx.new_file(path: Path.join(__DIR__, "../assets/transcribe.mp3"))
The file parameter is used to create the Audio.Transcription request structure
translation_req = Audio.Transcription.new(file: audio_file, model: "whisper-1")
We then call the Audio.Transcription.create()
function to create a transcription.
translation_response = openai |> Audio.Transcription.create(translation_req)
Create translation
The translation call uses practically the same request structure, but calls the Audio.Translation.create()
endpoint
For more information on the audio endpoints see the Openai API Audio Reference
File
List files
To request all files that belong to the user organization, call the File.list()
function
alias OpenaiEx.File
openai |> File.list()
Upload files
To upload a file, we need to create a file parameter, and then the upload request
# if you're having problems downloading raw github content, you may need to manually set your DNS server to "8.8.8.8" (google)
ftf_url = "https://raw.githubusercontent.com/restlessronin/openai_ex/main/assets/fine-tune.jsonl"
fine_tune_file = OpenaiEx.new_file(name: ftf_url, content: fetch_blob.(ftf_url))
# fine_tune_file = OpenaiEx.new_file(path: Path.join(__DIR__, "../assets/fine-tune.jsonl"))
upload_req = File.new_upload(file: fine_tune_file, purpose: "fine-tune")
Then we call the File.create()
function to upload the file
upload_res = openai |> File.create(upload_req)
We can verify that the file has been uploaded by calling
openai |> File.list()
We grab the file id from the previous response value to use in the following samples
file_id = upload_res["id"]
Retrieve files
In order to retrieve meta information on a file, we simply call the File.retrieve()
function with the given id
openai |> File.retrieve(file_id)
Retrieve file content
Similarly to download the file contents, we call File.download()
openai |> File.download(file_id)
Delete file
Finally, we can delete the file by calling File.delete()
openai |> File.delete(file_id)
Verify that the file has been deleted by listing files again
openai |> File.list()
FineTuning Job
To run a fine-tuning job, we minimally need a training file. We will re-run the file creation request above.
upload_res = openai |> File.create(upload_req)
Next we call FineTuning.Job.new()
to create a new request structure
alias OpenaiEx.FineTuning
ft_req = FineTuning.Job.new(model: "davinci-002", training_file: upload_res["id"])
To begin the fine tune, we call the FineTune.create()
function
ft_res = openai |> FineTuning.Job.create(ft_req)
We can list all fine tunes by calling FineTune.list()
openai |> FineTuning.Job.list()
The function FineTune.retrieve()
gets the details of a particular fine tune.
ft_id = ft_res["id"]
openai |> FineTuning.Job.retrieve(fine_tuning_job_id: ft_id)
and FineTune.list_events()
can be called to get the events
openai |> FineTuning.Job.list_events(fine_tuning_job_id: ft_id)
To cancel a Fine Tune job, call FineTune.cancel()
openai |> FineTuning.Job.cancel(fine_tuning_job_id: ft_id)
A fine tuned model can be deleted by calling the Model.delete()
ft_model = ft_res["fine_tuned_model"]
unless is_nil(ft_model) do
openai |> Model.delete(ft_model)
end
For more information on the fine tune endpoints see the Openai API Moderation Reference
Moderation
We use the moderation API by calling Moderation.new()
to create a new request
alias OpenaiEx.Moderation
mod_req = Moderation.new(input: "I want to kill people")
The call the function Moderation.create()
mod_res = openai |> Moderation.create(mod_req)
For more information on the moderation endpoints see the Openai API Moderation Reference
Assistant
alias OpenaiEx.Beta.Assistant
Create Assistant
To create an assistant with model and instructions, call the Assistant.create()
function.
First, we setup the create request parameters. This request sets up an Assistant with a code interpreter tool.
math_assistant_req =
Assistant.new(
instructions:
"You are a personal math tutor. When asked a question, write and run Python code to answer the question.",
name: "Math Tutor",
tools: [%{type: "code_interpreter"}],
model: "gpt-4"
)
Then we call the create function
asst = openai |> Assistant.create(math_assistant_req)
Retrieve Assistant
Extract the id field for the assistant
assistant_id = asst["id"]
which can then be used to retrieve the Assistant fields, using the Assistant.retrieve()
function.
openai |> Assistant.retrieve(assistant_id)
Modify Assistant
Once created, an assistant can be modified using the Assistant.update()
function.
Now we will show an example assistant request using the retrieval tool with a set of files. First we set up the files (in this case a sample HR document) by uploading using the File
API.
alias OpenaiEx.File
hr_file = OpenaiEx.new_file(path: Path.join(__DIR__, "../assets/cyberdyne.txt"))
hr_upload_req = File.new_upload(file: hr_file, purpose: "assistants")
hr_upload_res = openai |> File.create(hr_upload_req)
file_id = hr_upload_res["id"]
Next we create the update request
hr_assistant_req =
Assistant.new(
instructions:
"You are an HR bot, and you have access to files to answer employee questions about company policies. Always respond with info from one of the files.",
name: "HR Helper",
tools: [%{type: "retrieval"}],
model: "gpt-3.5-turbo-1106",
file_ids: [file_id]
)
Finally we call the endpoint to modify the Assistant
asst = openai |> Assistant.update(assistant_id, hr_assistant_req)
Delete Assistant
Finally we can delete assistants using the Assistant.delete()
function
openai |> Assistant.delete(assistant_id)
List Assistants
We use Assistant.list()
to get a list of assistants
assts = openai |> Assistant.list()
Create Assistant File
Use Assistants.File.create()
to attach a file to an assistant.
Let us re-create the first assistant above
math_asst = openai |> Assistant.create(math_assistant_req)
Now we can attach the file that we used earlier to this new assistant.
math_assistant_id = math_asst["id"]
asst_f = Assistant.File.new(assistant_id: math_assistant_id, file_id: file_id)
asst_file = openai |> Assistant.File.create(asst_f)
Let's retrieve the assistant to see if it was updated with the file id (check file_ids
field value).
openai |> Assistant.retrieve(math_assistant_id)
Retreive Assistant File
We can retrieve an assistant file by using Assistants.File.retrieve()
openai |> Assistant.File.retrieve(asst_f)
Delete Assistant File
[Assistant.File.delete()
] can be used to detach a file from the assistant.
openai |> Assistant.File.delete(asst_f)
Verify detach by retrieving the assistant again (check file_ids
field value)
openai |> Assistant.retrieve(math_assistant_id)
List Assistant Files
We can list all files attached to an assistant with Assistant.File.list()
list_req = Assistant.File.new_list(assistant_id: math_assistant_id)
openai |> Assistant.File.list(list_req)
Thread
alias OpenaiEx.Beta.Thread
Create thread
Use the [Thread.create()
] function to create threads. A thread can be created empty or with messages.
alias OpenaiEx.ChatMessage
empty_thread = openai |> Thread.create()
msg_hr = ChatMessage.user("What company do we work at?", [file_id])
msg_ai = ChatMessage.user("How does AI work? Explain it in simple terms.")
thrd_req = Thread.new(messages: [msg_hr, msg_ai])
thread = openai |> Thread.create(thrd_req)
Retrieve thread
Thread.retrieve()
can be used to get the thread parameters given the id.
thread_id = thread["id"]
openai |> Thread.retrieve(thread_id)
Modify thread
The metadata for a thread can be modified using Thread.update()
openai |> Thread.update(thread_id, %{metadata: %{modified: "true", user: "abc123"}})
Delete thread
Use Thread.delete()
to delete a thread
openai |> Thread.delete(thread_id)
Verify deletion
openai |> Thread.retrieve(thread_id)
Messages
alias OpenaiEx.Beta.Thread.Message
Create message
You can create a single message for a thread using Message.create()
thread_id = empty_thread["id"]
message = openai |> Message.create(thread_id, msg_hr)
Retrieve message
Use [Message.retrieve()
] to retrieve a message
message_id = message["id"]
openai |> Message.retrieve(%{thread_id: thread_id, message_id: message_id})
Modify message
The metadata for a message can be modified by [Message.update()
]
metadata = %{modified: "true", user: "abc123"}
upd_msg_req = Message.new(thread_id: thread_id, message_id: message_id, metadata: metadata)
message = openai |> Message.update(upd_msg_req)
List messages
Use [Message.list()
] to get all the messages for a given thread
openai |> Message.list(thread_id)
Retrieve message file
Retrieve a message file using [Message.File.retrieve()
]
openai |> Message.File.retrieve(%{thread_id: thread_id, message_id: message_id, file_id: file_id})
List message files
You can list all message files using [Message.File.list()
]
openai |> Message.File.list(%{thread_id: thread_id, message_id: message_id})
Runs
alias OpenaiEx.Beta.Thread.Run
Create run
A run represents an execution on a thread. Use to Run.create()
with an assistant on a thread
run_req = Run.new(thread_id: thread_id, assistant_id: math_assistant_id)
run = openai |> Run.create(run_req)
Retrieve run
Retrieve a run using Run.retrieve()
run_id = run["id"]
openai |> Run.retrieve(%{thread_id: thread_id, run_id: run_id})
Modify run
The run metadata can be modified using the Run.update()
function
openai
|> Run.update(%{
thread_id: thread_id,
run_id: run_id,
metadata: %{user_id: "user_zmVY6FvuBDDwIqM4KgH"}
})
List runs
List the runs belonging to a thread using Run.list()
openai |> Run.list(thread_id)
Submit tool outputs to a run
When a run has the status
: "requires_action" and required_action.type
is submit_tool_outputs
, the Run.submit_tool_outputs()
can be used to submit the outputs from the tool calls once they're all completed. All outputs must be submitted in a single request.
openai
|> Run.submit_tool_outputs(%{
thread_id: thread_id,
run_id: run_id,
tool_outputs: [%{tool_call_id: "foobar", output: "28C"}]
})
Cancel a run
You can cancel a run in_progress
using Run.cancel()
openai |> Run.cancel(%{thread_id: thread_id, run_id: run_id})
Create thread and run
Use Run.create_and_run()
to create a thread and run.