View Source LangChain.Message.ContentPart (LangChain v0.4.0-rc.2)

Models a ContentPart. ContentParts are now used for multi-modal support in both messages and tool results. This enables richer responses, allowing text, images, files, and thinking blocks to be combined in a single message or tool result.

Types

:text - The message part is text.
:image_url - The message part is a URL to an image.
:image - The message part is image data that is base64 encoded text.
:file - The message part is file data that is base64 encoded text.
:file_url - The message part is a URL to a file.
:thinking - A thinking block from a reasoning model like Anthropic.
:unsupported - A part that is not supported but may need to be present. This includes Anthropic's redacted_thinking block which has no value in being displayed because it is encrypted, but can be provided back to the LLM to maintain reasoning continuity. The specific parts of the data are stored in :options.

Fields

:content - Text content.
:options - Options are a keyword list of values that may be specific to the LLM for a particular message type. For example, multi-modal message (ones that include image data) use the :media option to specify the mimetype information. Options may also contain key-value settings like cache_control: true for models like Anthropic that support caching.
When receiving content parts like with Anthropic Claude's thinking model, the options may contain LLM specific data that is recommended to be preserved like a signature or redacted_thinking data used by the LLM.

Image mime types

The :media option is used to specify the mime type of the image. Various LLMs handle this differently or perhaps not at all.

Examples:

media: :jpg - turns into "image/jpeg" or "image/jpg", depending on what the LLM accepts.
media: :png - turns into "image/png"
media: "image/webp" - stays as "image/webp". Any specified string value is passed through unchanged. This allows for future formats to be supported quickly.
When omitted, the LLM may error or some will accept it but may require the base64 encoded content data to be prefixed with the mime type information. Basically, you must handle the content needs yourself.

Summary

Types

t()

Functions

content_to_string(content, type \\ :text)

Convert "content" to a string. Content may be nil, a string, or a list of ContentParts.

file!(content, opts \\ [])

Create a new ContentPart that contains a file encoded as base64 data.

file_url!(content, opts \\ [])

Create a new ContentPart that contains a URL to an file. Raises an exception if not valid.

image!(content, opts \\ [])

Create a new ContentPart that contains an image encoded as base64 data. Raises an exception if not valid.

image_url!(content, opts \\ [])

Create a new ContentPart that contains a URL to an image. Raises an exception if not valid.

merge_part(primary, new_part)

Merge two ContentPart structs for the same index in a MessageDelta. The first ContentPart is the primary one that smaller deltas are merged into. The primary is what is being accumulated.

new(attrs \\ %{})

Build a new message and return an :ok/:error tuple with the result.

new!(attrs \\ %{})

Build a new message and return it or raise an error if invalid.

parts_to_string(parts, type \\ :text)

Helper function for easily getting plain text from a list of ContentParts.

set_option_on_last_part(content_parts, option_key, option_value)

Sets an option on the last text part in a list of ContentParts. Returns the updated content parts.

text!(content, opts \\ [])

Create a new ContentPart that contains text. Raises an exception if not valid.

Types

t()

@type t() :: %LangChain.Message.ContentPart{
  content: term(),
  options: term(),
  type: term()
}

Functions

content_to_string(content, type \\ :text)

@spec content_to_string(content :: String.t() | [t()] | nil, type :: atom()) ::
  nil | String.t()

Convert "content" to a string. Content may be nil, a string, or a list of ContentParts.

file!(content, opts \\ [])

@spec file!(String.t(), Keyword.t()) :: t() | no_return()

Create a new ContentPart that contains a file encoded as base64 data.

file_url!(content, opts \\ [])

@spec file_url!(String.t(), Keyword.t()) :: t() | no_return()

Create a new ContentPart that contains a URL to an file. Raises an exception if not valid.

image!(content, opts \\ [])

@spec image!(String.t(), Keyword.t()) :: t() | no_return()

Create a new ContentPart that contains an image encoded as base64 data. Raises an exception if not valid.

Options

:media - Provide the "media type" for the image. Examples: "image/jpeg", "image/png", etc. ChatGPT does not require this but other LLMs may.
:detail - if the LLM supports it, most images must be resized or cropped before given to the LLM for analysis. A detail option may specify the level detail of the image to present to the LLM. The higher the detail, the more tokens consumed. Currently only supported by OpenAI and the values of "low", "high", and "auto".

ChatGPT requires media type information to prefix the base64 content. Setting the media: "image/jpeg" type will do that. Otherwise the data must be provided with the required prefix.

Anthropic requires the media type information to be submitted as separate information with the JSON request. This media option provides an abstraction to normalize the behavior.

image_url!(content, opts \\ [])

@spec image_url!(String.t(), Keyword.t()) :: t() | no_return()

Create a new ContentPart that contains a URL to an image. Raises an exception if not valid.

merge_part(primary, new_part)

@spec merge_part(nil | t(), t()) :: t()

Merge two ContentPart structs for the same index in a MessageDelta. The first ContentPart is the primary one that smaller deltas are merged into. The primary is what is being accumulated.

A set of ContentParts can be merged like this:

Enum.reduce(list_of_content_parts, nil, fn new_part, acc ->
  ContentPart.merge_part(acc, new_part)
end)

new(attrs \\ %{})

@spec new(attrs :: map()) :: {:ok, t()} | {:error, Ecto.Changeset.t()}

Build a new message and return an :ok/:error tuple with the result.

new!(attrs \\ %{})

@spec new!(attrs :: map()) :: t() | no_return()

Build a new message and return it or raise an error if invalid.

Example

ContentPart.new!(%{type: :text, content: "Greetings!"})

ContentPart.new!(%{type: :image_url, content: "https://example.com/images/house.jpg"})

ContentPart.new!(%{type: :thinking, content: "I've been asked...", options: [signature: "SIGNATURE_DATA"]}

ContentPart.new!(%{type: :unsupported, content: "redacted_data", options: [type: "redacted_thinking"]}

parts_to_string(parts, type \\ :text)

@spec parts_to_string([t()], type :: atom()) :: nil | String.t()

Helper function for easily getting plain text from a list of ContentParts.

This function processes a list of ContentParts and joins the text parts together using "

" characters. Only parts where type: :text are used. All other parts are ignored.

Examples

iex> parts = [
...>   text!("Hello"),
...>   image!("base64data"),
...>   text!("world")
...> ]
iex> parts_to_string(parts)
"Hello\n\nworld"

iex> parts_to_string([])
nil

set_option_on_last_part(content_parts, option_key, option_value)

@spec set_option_on_last_part([t()], atom(), any()) :: [t()]

Sets an option on the last text part in a list of ContentParts. Returns the updated content parts.

text!(content, opts \\ [])

Create a new ContentPart that contains text. Raises an exception if not valid.