View Source InstructorLite.Adapters.Llamacpp (instructor_lite v0.3.0)

LLaMA.cpp HTTP Server adapter.

This adapter is implemented using llama-specific completion endpoint.

Params

params argument should be shaped as a completion request body

Example 

InstructorLite.instruct(%{
    prompt: "John is 25yo",
    temperature: 0.8
  },
  response_model: %{name: :string, age: :integer},
  adapter: InstructorLite.Adapters.Llamacpp,
  adapter_context: [url: "http://localhost:8000/completion"]
)
{:ok, %{name: "John", age: 25}}

Summary

Functions

initial_prompt(params, opts)

Updates params with prompt based on json_schema and notes.

parse_response(response, opts)

Parse API response.

retry_prompt(params, resp_params, errors, response, opts)

Updates params with prompt for retrying a request.

send_request(params, opts)

Make request to llamacpp HTTP server.

Functions

initial_prompt(params, opts)

Updates params with prompt based on json_schema and notes.

It uses json_schema and system_prompt request parameters.

parse_response(response, opts)

Parse API response.

Can return:

  • {:ok, parsed_json} on success.
  • {:error, :unexpected_response, response} if response is of unexpected shape.

retry_prompt(params, resp_params, errors, response, opts)

Updates params with prompt for retrying a request.

send_request(params, opts)

Make request to llamacpp HTTP server.

Options

  • :http_client (atom/0) - Any module that follows Req.post/2 interface The default value is Req.

  • :http_options (keyword/0) - Options passed to http_client.post/2 The default value is [receive_timeout: 60000].

  • :url (String.t/0) - Required. API endpoint to use, for example http://localhost:8000/completion