View Source LangChain.Chains.LLMChain (LangChain v0.3.3)
Summary
Types
A message processor is an arity 2 function that takes an LLMChain and a Message. It is used to "pre-process" the received message from the LLM. Processors can be chained together to perform a sequence of transformations.
The expected return types for a Message processor function. When successful,
it returns a :continue
with an Message to use as a replacement. When it
fails, a :halt
is returned along with an updated LLMChain.t()
and a new
user message to be returned to the LLM reporting the error.
Functions
Define an LLMChain. This is the heart of the LangChain library.
Add another callback to the list of callbacks.
Add a received Message struct to the chain. The LLMChain tracks the
last_message
received and the complete list of messages exchanged. Depending
on the message role, the chain may be in a pending or incomplete state where
a response from the LLM is anticipated.
Add a set of Message structs to the chain. This enables quickly building a chain for submitting to an LLM.
Add a tool to an LLMChain.
Apply a received MessageDelta struct to the chain. The LLMChain tracks the
current merged MessageDelta state. When the final delta is received that
completes the message, the LLMChain is updated to clear the delta
and the
last_message
and list of messages are updated.
Apply a list of deltas to the chain.
Apply a set of PromptTemplates to the chain. The list of templates can also include Messages with no templates. Provide the inputs to apply to the templates for rendering as a message. The prepared messages are applied to the chain.
Remove an incomplete MessageDelta from delta
and add a Message with the
desired status to the chain.
Convert any hanging delta of the chain to a message and append to the chain.
Execute the tool call with the tool. Returns the tool's message response.
If the last_message
from the Assistant includes one or more ToolCall
s, then the linked
tool is executed. If there is no last_message
or the last_message
is
not a tool_call
, the LLMChain is returned with no action performed.
This makes it safe to call any time.
Increments the internal current_failure_count. Returns and incremented and updated struct.
Register a set of processors to on received assistant messages.
Start a new LLMChain configuration.
Start a new LLMChain configuration and return it or raise an error if invalid.
Process a newly message received from the LLM. Messages with a role of
:assistant
may be processed through the message_processors
before being
generally available or being notified through a callback.
Convenience function for setting the prompt text for the LLMChain using prepared text.
Reset the internal current_failure_count to 0. Useful after receiving a successfully returned and processed message from the LLM.
Reset the internal current_failure_count to 0 if the function provided returns
true
. Helps to make the change conditional.
Run the chain on the LLM using messages and any registered functions. This formats the request for a ChatLLMChain where messages are passed to the API.
Run the chain until a specific tool call is made. This makes it easy for an LLM to make multiple tool calls and call a specific tool to return a result, signaling the end of the operation.
Update the LLMChain's custom_context
map. Passing in a context_update
map
will by default merge the map into the existing custom_context
.
Types
@type message_processor() :: (t(), LangChain.Message.t() -> processor_return())
A message processor is an arity 2 function that takes an LLMChain and a Message. It is used to "pre-process" the received message from the LLM. Processors can be chained together to perform a sequence of transformations.
@type processor_return() :: {:continue, LangChain.Message.t()} | {:halt, t(), LangChain.Message.t()}
The expected return types for a Message processor function. When successful,
it returns a :continue
with an Message to use as a replacement. When it
fails, a :halt
is returned along with an updated LLMChain.t()
and a new
user message to be returned to the LLM reporting the error.
@type t() :: %LangChain.Chains.LLMChain{ _tool_map: term(), callbacks: term(), current_failure_count: term(), custom_context: term(), delta: term(), exchanged_messages: term(), last_message: term(), llm: term(), max_retry_count: term(), message_processors: term(), messages: term(), needs_response: term(), tools: term(), verbose: term(), verbose_deltas: term() }
Functions
Define an LLMChain. This is the heart of the LangChain library.
The chain deals with tools, a tool map, delta tracking, tracking the messages exchanged during a run, the last_message tracking, conversation messages, and verbose logging. This helps by separating these responsibilities from the LLM making it easier to support additional LLMs because the focus is on communication and formats instead of all the extra logic.
Callbacks
Callbacks are fired as specific events occur in the chain as it is running.
The set of events are defined in LangChain.Chains.ChainCallbacks
.
To be notified of an event you care about, register a callback handler with
the chain. Multiple callback handlers can be assigned. The callback handler
assigned to the LLMChain
is not provided to an LLM chat model. For callbacks
on a chat model, set them there.
Registering a callback handler
A handler is a map with key name for the callback to fire. A function is assigned to the map key. Refer to the documentation for each function as they arguments vary.
If we want to be notified when an LLM Assistant chat response message has been processed and it is complete, this is how we could receive that event in our running LiveView:
live_view_pid = self()
handler = %{
on_message_processed: fn _chain, message ->
send(live_view_pid, {:new_assistant_response, message})
end
}
LLMChain.new!(%{...})
|> LLMChain.add_callback(handler)
|> LLMChain.run()
In the LiveView, a handle_info
function executes with the received message.
Fallbacks
When running a chain, the :with_fallbacks
option can be used to provide a
list of fallback chat models to try when a failure is encountered.
When working with language models, you may often encounter issues from the underlying APIs, whether these be rate limiting, downtime, or something else. Therefore, as you go to move your LLM applications into production it becomes more and more important to safeguard against these. That's what fallbacks are designed to provide.
A fallback is an alternative plan that may be used in an emergency.
A before_fallback
function can be provided to alter or return a different
chain to use with the fallback LLM model. This is important because often, the
prompts needed for will differ for a fallback LLM. This means if your OpenAI
completion fails, a different prompt may be needed when retrying it with an
Anthropic fallback.
Fallback for LLM API Errors
This is perhaps the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of failures.
Fallback Examples
A simple fallback that tries a different LLM chat model
fallback_llm = ChatAnthropic.new!(%{stream: false})
{:ok, updated_chain} =
%{llm: ChatOpenAI.new!(%{stream: false})}
|> LLMChain.new!()
|> LLMChain.add_message(Message.new_system!("OpenAI system prompt"))
|> LLMChain.add_message(Message.new_user!("Why is the sky blue?"))
|> LLMChain.run(with_fallbacks: [fallback_llm])
Note the with_fallbacks: [fallback_llm]
option when running the chain.
This example uses the :before_fallback
option to provide a function that can
modify or return an alternate chain when used with a certain LLM. Also note
the utility function LangChain.Utils.replace_system_message!/2
is used for
swapping out the system message when falling back to a different LLM.
fallback_llm = ChatAnthropic.new!(%{stream: false})
{:ok, updated_chain} =
%{llm: ChatOpenAI.new!(%{stream: false})}
|> LLMChain.new!()
|> LLMChain.add_message(Message.new_system!("OpenAI system prompt"))
|> LLMChain.add_message(Message.new_user!("Why is the sky blue?"))
|> LLMChain.run(
with_fallbacks: [fallback_llm],
before_fallback: fn chain ->
case chain.llm do
%ChatAnthropic{} ->
# replace the system message
%LLMChain{
chain
| messages:
Utils.replace_system_message!(
chain.messages,
Message.new_system!("Anthropic system prompt")
)
}
_open_ai ->
chain
end
end
)
See LangChain.Chains.LLMChain.run/2
for more details.
Run Until Tool Used
The run_until_tool_used/3
function makes it easy to instruct an LLM to use a
set of tools and then call a specific tool to present the results. This is
particularly useful for complex workflows where you want the LLM to perform
multiple operations and then finalize with a specific action.
This works well for receiving a final structured output after multiple tools are used.
When the specified tool is successfully called, the chain stops processing and returns the result. This prevents unnecessary additional LLM calls and provides a clear termination point for your workflow.
{:ok, %LLMChain{} = updated_chain, %ToolResult{} = tool_result} =
%{llm: ChatOpenAI.new!(%{stream: false})}
|> LLMChain.new!()
|> LLMChain.add_tools([special_search, report_results])
|> LLMChain.add_message(Message.new_system!())
|> LLMChain.add_message(Message.new_user!("..."))
|> LLMChain.run_until_tool_used("final_summary")
The function returns a tuple with three elements:
:ok
- Indicating success- The updated chain with all messages and tool calls
- The specific tool result that matched the requested tool name
To prevent runaway function calls, a default max_runs
value of 25 is set.
You can adjust this as needed:
# Allow up to 50 runs before timing out
LLMChain.run_until_tool_used(chain, "final_summary", max_runs: 50)
The function also supports fallbacks, allowing you to gracefully handle LLM failures:
LLMChain.run_until_tool_used(chain, "final_summary",
max_runs: 10,
with_fallbacks: [fallback_llm],
before_fallback: fn chain ->
# Modify chain before using fallback LLM
chain
end
)
See LangChain.Chains.LLMChain.run_until_tool_used/3
for more details.
@spec add_callback(t(), LangChain.Chains.ChainCallbacks.chain_callback_handler()) :: t()
Add another callback to the list of callbacks.
@spec add_message(t(), LangChain.Message.t()) :: t()
Add a received Message struct to the chain. The LLMChain tracks the
last_message
received and the complete list of messages exchanged. Depending
on the message role, the chain may be in a pending or incomplete state where
a response from the LLM is anticipated.
@spec add_messages(t(), [LangChain.Message.t()]) :: t()
Add a set of Message structs to the chain. This enables quickly building a chain for submitting to an LLM.
@spec add_tools( t(), LangChain.NativeTool.t() | LangChain.Function.t() | [LangChain.Function.t()] ) :: t() | no_return()
Add a tool to an LLMChain.
@spec apply_delta( t(), LangChain.MessageDelta.t() | {:error, LangChain.LangChainError.t()} ) :: t()
Apply a received MessageDelta struct to the chain. The LLMChain tracks the
current merged MessageDelta state. When the final delta is received that
completes the message, the LLMChain is updated to clear the delta
and the
last_message
and list of messages are updated.
Apply a list of deltas to the chain.
@spec apply_prompt_templates( t(), [LangChain.Message.t() | LangChain.PromptTemplate.t()], %{ required(atom()) => any() } ) :: t() | no_return()
Apply a set of PromptTemplates to the chain. The list of templates can also include Messages with no templates. Provide the inputs to apply to the templates for rendering as a message. The prepared messages are applied to the chain.
Remove an incomplete MessageDelta from delta
and add a Message with the
desired status to the chain.
Convert any hanging delta of the chain to a message and append to the chain.
If the delta is nil
, the chain is returned unmodified.
@spec execute_tool_call( LangChain.Message.ToolCall.t(), LangChain.Function.t(), Keyword.t() ) :: LangChain.Message.ToolResult.t()
Execute the tool call with the tool. Returns the tool's message response.
If the last_message
from the Assistant includes one or more ToolCall
s, then the linked
tool is executed. If there is no last_message
or the last_message
is
not a tool_call
, the LLMChain is returned with no action performed.
This makes it safe to call any time.
The context
is additional data that will be passed to the executed tool.
The value given here will override any custom_context
set on the LLMChain.
If not set, the global custom_context
is used.
Increments the internal current_failure_count. Returns and incremented and updated struct.
@spec message_processors(t(), [message_processor()]) :: t()
Register a set of processors to on received assistant messages.
@spec new(attrs :: map()) :: {:ok, t()} | {:error, Ecto.Changeset.t()}
Start a new LLMChain configuration.
{:ok, chain} = LLMChain.new(%{
llm: %ChatOpenAI{model: "gpt-3.5-turbo", stream: true},
messages: [%Message.new_system!("You are a helpful assistant.")]
})
Start a new LLMChain configuration and return it or raise an error if invalid.
chain = LLMChain.new!(%{
llm: %ChatOpenAI{model: "gpt-3.5-turbo", stream: true},
messages: [%Message.new_system!("You are a helpful assistant.")]
})
@spec process_message(t(), LangChain.Message.t()) :: t()
Process a newly message received from the LLM. Messages with a role of
:assistant
may be processed through the message_processors
before being
generally available or being notified through a callback.
Convenience function for setting the prompt text for the LLMChain using prepared text.
Reset the internal current_failure_count to 0. Useful after receiving a successfully returned and processed message from the LLM.
Reset the internal current_failure_count to 0 if the function provided returns
true
. Helps to make the change conditional.
@spec run(t(), Keyword.t()) :: {:ok, t()} | {:error, t(), LangChain.LangChainError.t()}
Run the chain on the LLM using messages and any registered functions. This formats the request for a ChatLLMChain where messages are passed to the API.
When successful, it returns {:ok, updated_chain}
Options
:mode
- It defaults to run the chain one time, stopping after receiving a response from the LLM. Supports:until_success
and:while_needs_response
.mode: :until_success
- (for non-interactive processing done by the LLM where it may repeatedly fail and need to re-try) Repeatedly evaluates a received message through any message processors, returning any errors to the LLM until it either succeeds or exceeds themax_retry_count
. This includes evaluating receivedToolCall
s until they succeed. If an LLM makes 3 ToolCalls in a single message and 2 succeed while 1 fails, the success responses are returned to the LLM with the failure response of the remainingToolCall
, giving the LLM an opportunity to resend the failedToolCall
, and only the failedToolCall
until it succeeds or exceeds themax_retry_count
. In essence, once we have a successful response from the LLM, we don't return any more to it and don't want any further responses.mode: :while_needs_response
- (for interactive chats that makeToolCalls
) Repeatedly evaluates functions and submits to the LLM so long as we still expect to get a response. Best fit for conversational LLMs where aToolResult
is used by the LLM to continue. After allToolCall
messages are evaluated, theToolResult
messages are returned to the LLM giving it an opportunity to use theToolResult
information in an assistant response message. In essence, this mode always gives the LLM the last word.with_fallbacks: [...]
- Provide a list of chat models to use as a fallback when one fails. This helps a production system remain operational when an API limit is reached, an LLM service is overloaded or down, or something else new an exciting goes wrong.When all fallbacks fail, a
%LangChainError{type: "all_fallbacks_failed"}
is returned in the error response.before_fallback: fn chain -> modified_chain end
- Abefore_fallback
function is called before the LLM call is made. NOTE: When provided, it also fires for the first attempt. This allows a chain to be modified or replaced before running against the configured LLM. This is helpful, for example, when a different system prompt is needed for Anthropic vs OpenAI.
Mode Examples
Use Case: A chat with an LLM where functions are available to the LLM:
LLMChain.run(chain, mode: :while_needs_response)
This will execute any LLM called functions, returning the result to the LLM, and giving it a chance to respond to the results.
Use Case: An application that exposes a function to the LLM, but we want to stop once the function is successfully executed. When errors are encountered, the LLM should be given error feedback and allowed to try again.
LLMChain.run(chain, mode: :until_success)
Run the chain until a specific tool call is made. This makes it easy for an LLM to make multiple tool calls and call a specific tool to return a result, signaling the end of the operation.
Options
max_runs
: The maximum number of times to run the chain. To prevent runaway calls, it defaults to 25. When exceeded, a%LangChainError{type: "exceeded_max_runs"}
is returned in the error response.with_fallbacks: [...]
- Provide a list of chat models to use as a fallback when one fails. This helps a production system remain operational when an API limit is reached, an LLM service is overloaded or down, or something else new an exciting goes wrong.When all fallbacks fail, a
%LangChainError{type: "all_fallbacks_failed"}
is returned in the error response.before_fallback: fn chain -> modified_chain end
- Abefore_fallback
function is called before the LLM call is made. NOTE: When provided, it also fires for the first attempt. This allows a chain to be modified or replaced before running against the configured LLM. This is helpful, for example, when a different system prompt is needed for Anthropic vs OpenAI.
@spec update_custom_context( t(), context_update :: %{required(atom()) => any()}, opts :: Keyword.t() ) :: t() | no_return()
Update the LLMChain's custom_context
map. Passing in a context_update
map
will by default merge the map into the existing custom_context
.
Use the :as
option to:
:merge
- Merge update changes in. Default.:replace
- Replace the context with thecontext_update
.