req
v0.4.0

View Source Req.Steps (req v0.4.0)

The collection of built-in steps.

Req is composed of three main pieces:

  • Req - the high-level API

  • Req.Request - the low-level API and the request struct

  • Req.Steps - the collection of built-in steps (you're here!)

Summary

Request Steps

auth(request)

Sets request authentication.

cache(request)

Performs HTTP caching using if-modified-since header.

compress_body(request)

Compresses the request body.

compressed(request)

Asks the server to return compressed response.

encode_body(request)

Encodes the request body.

put_base_url(request)

Sets base URL for all requests.

put_params(request)

Adds params to request query string.

put_path_params(request)

Uses a templated request path.

put_plug(request)

Runs the request against a plug instead of over the network.

put_range(request)

Sets the "Range" request header.

put_user_agent(request)

Sets the user-agent header.

run_finch(request)

Runs the request using Finch.

Response Steps

decode_body(request_response)

Decodes response body based on the detected format.

decompress_body(request_response)

Decompresses the response body based on the content-encoding header.

handle_http_errors(request_response)

Handles HTTP 4xx/5xx error responses.

redirect(request_response)

Follows redirects.

Error Steps

retry(request_response_or_error)

Retries a request in face of errors.

Request Steps

Link to this function

auth(request)

View Source

Sets request authentication.

Request Options

  • :auth - sets the authorization header:

    • string - sets to this value;

    • {username, password} - uses Basic HTTP authentication;

    • {:bearer, token} - uses Bearer HTTP authentication;

    • :netrc - load credentials from .netrc at path specified in NETRC environment variable. If NETRC is not set, load .netrc in user's home directory;

    • {:netrc, path} - load credentials from path

Examples 

iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {"bad", "bad"}).status
401
iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {"foo", "bar"}).status
200

iex> Req.get!("https://httpbin.org/bearer", auth: {:bearer, ""}).status
401
iex> Req.get!("https://httpbin.org/bearer", auth: {:bearer, "foo"}).status
200

iex> System.put_env("NETRC", "./test/my_netrc")
iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: :netrc).status
200

iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {:netrc, "./test/my_netrc"}).status
200
Link to this function

cache(request)

View Source

Performs HTTP caching using if-modified-since header.

Only successful (200 OK) responses are cached.

This step also prepends a response step that loads and writes the cache. Be careful when prepending other response steps, make sure the cache is loaded/written as soon as possible.

Options

  • :cache - if true, performs simple caching using if-modified-since header. Defaults to false.

  • :cache_dir - the directory to store the cache, defaults to <user_cache_dir>/req (see: :filename.basedir/3)

Examples 

iex> url = "https://elixir-lang.org"
iex> response1 = Req.get!(url, cache: true)
iex> response2 = Req.get!(url, cache: true)
iex> response1 == response2
true
Link to this function

compress_body(request)

View Source

Compresses the request body.

Request Options

  • :compress_body - if set to true, compresses the request body using gzip. Defaults to false.
Link to this function

compressed(request)

View Source

Asks the server to return compressed response.

Supported formats:

  • gzip
  • br (if brotli is installed)
  • zstd (if ezstd is installed)

Request Options

  • :compressed - if set to true, sets the accept-encoding header with compression algorithms that Req supports. Defaults to true.

Examples

Req automatically decompresses response body (decompress_body/1 step) so let's disable that by passing raw: true.

By default, we ask the server to send compressed response. Let's look at the headers and the raw body. Notice the body starts with <<31, 139>> (<<0x1F, 0x8B>>), the "magic bytes" for gzip:

iex> response = Req.get!("https://elixir-lang.org", raw: true)
iex> Req.Response.get_header(response, "content-encoding")
["gzip"]
iex> response.body |> binary_part(0, 2)
<<31, 139>>

Now, let's pass compressed: false and notice the raw body was not compressed:

iex> response = Req.get!("https://elixir-lang.org", raw: true, compressed: false)
iex> Req.Response.get_header(response, "content-encoding")
nil
iex> response.body |> binary_part(0, 15)
"<!DOCTYPE html>"

The Brotli and Zstandard compression algorithms are also supported if the optional packages are installed:

Mix.install([
  :req,
  {:brotli, "~> 0.3.0"},
  {:ezstd, "~> 1.0"}
])

response = Req.get!("https://httpbin.org/anything")
response.body["headers"]["Accept-Encoding"]
#=> "zstd, br, gzip"
Link to this function

encode_body(request)

View Source

Encodes the request body.

Request Options

  • :form - if set, encodes the request body as form data (using URI.encode_query/1).

  • :json - if set, encodes the request body as JSON (using Jason.encode_to_iodata!/1), sets the accept header to application/json, and the content-type header to application/json.

Examples 

iex> Req.post!("https://httpbin.org/anything", form: [x: 1]).body["form"]
%{"x" => "1"}

iex> Req.post!("https://httpbin.org/post", json: %{x: 2}).body["json"]
%{"x" => 2}
Link to this function

put_base_url(request)

View Source

Sets base URL for all requests.

Request Options

  • :base_url - if set, the request URL is merged with this base URL.

Examples 

iex> req = Req.new(base_url: "https://httpbin.org")
iex> Req.get!(req, url: "/status/200").status
200
iex> Req.get!(req, url: "/status/201").status
201
Link to this function

put_params(request)

View Source

Adds params to request query string.

Request Options

  • :params - params to add to the request query string. Defaults to [].

Examples 

iex> Req.get!("https://httpbin.org/anything/query", params: [x: 1, y: 2]).body["args"]
%{"x" => "1", "y" => "2"}
Link to this function

put_path_params(request)

View Source

Uses a templated request path.

Request Options

  • :path_params - params to add to the templated path. Defaults to [].

Examples 

iex> Req.get!("https://httpbin.org/status/:code", path_params: [code: 200]).status
200
Link to this function

put_plug(request)

View Source

Runs the request against a plug instead of over the network.

Request Options

  • :plug - if set, the plug to run the request against.

Examples

This step is particularly useful to test plugs:

defmodule Echo do
  def call(conn, _) do
    "/" <> path = conn.request_path
    Plug.Conn.send_resp(conn, 200, path)
  end
end

test "echo" do
  assert Req.get!("http:///hello", plug: Echo).body == "hello"
end

You can define plugs as functions too:

test "echo" do
  echo = fn conn ->
    "/" <> path = conn.request_path
    Plug.Conn.send_resp(conn, 200, path)
  end

  assert Req.get!("http:///hello", plug: echo).body == "hello"
end

which is particularly useful to create HTTP service mocks with tools like Bypass.

Here is another example, let's run the request against Plug.Static pointed to the Req's source code and fetch the README:

iex> resp = Req.get!("http:///README.md", plug: {Plug.Static, at: "/", from: "."})
iex> resp.body =~ "Req is a batteries-included HTTP client for Elixir."
true
Link to this function

put_range(request)

View Source

Sets the "Range" request header.

Request Options

  • :range - can be one of the following:

    • a string - returned as is

    • a first..last range - converted to "bytes=<first>-<last>"

Examples 

iex> response = Req.get!("https://httpbin.org/range/100", range: 0..3)
iex> response.status
206
iex> response.body
"abcd"
iex> Req.Response.get_header(response, "content-range")
["bytes 0-3/100"]
Link to this function

put_user_agent(request)

View Source

Sets the user-agent header.

Request Options

  • :user_agent - sets the user-agent header. Defaults to "req/0.4.0".

Examples 

iex> Req.get!("https://httpbin.org/user-agent").body
%{"user-agent" => "req/0.4.0"}

iex> Req.get!("https://httpbin.org/user-agent", user_agent: "foo").body
%{"user-agent" => "foo"}
Link to this function

run_finch(request)

View Source

Runs the request using Finch.

This is the default Req adapter. See "Adapter" section in the Req.Request module documentation for more information on adapters.

Request Options

  • :finch - the name of the Finch pool. Defaults to a pool automatically started by Req. The default pool uses HTTP/1 although that may change in the future.

  • :connect_options - dynamically starts (or re-uses already started) Finch pool with the given connection options:

    • :timeout - socket connect timeout in milliseconds, defaults to 30_000.

    • :protocol - the HTTP protocol to use, defaults to :http1.

    • :hostname - Mint explicit hostname, see Mint.HTTP.connect/4 for more information.

    • :transport_opts - Mint transport options, see Mint.HTTP.connect/4 for more information.

    • :proxy_headers - Mint proxy headers, see Mint.HTTP.connect/4 for more information.

    • :proxy - Mint HTTP/1 proxy settings, a {schema, address, port, options} tuple. See Mint.HTTP.connect/4 for more information.

    • :client_settings - Mint HTTP/2 client settings, see Mint.HTTP.connect/4 for more information.

  • :inet6 - if set to true, uses IPv6. Defaults to false. This is a shortcut for setting connect_options: [transport_opts: [inet6: true]].

  • :pool_timeout - pool checkout timeout in milliseconds, defaults to 5000.

  • :receive_timeout - socket receive timeout in milliseconds, defaults to 15_000.

  • :unix_socket - if set, connect through the given UNIX domain socket.

  • :finch_request - a function that executes the Finch request, defaults to using Finch.request/3.

    The function should accept 4 arguments:

    • request - the %Req.Request{} struct

    • finch_request - the Finch request

    • finch_name - the Finch name

    • finch_options - the Finch options

    And it should return either {request, response} or {request, exception}.

Examples

Custom :receive_timeout:

iex> Req.get!(url, receive_timeout: 1000)

Connecting through UNIX socket:

iex> Req.get!("http:///v1.41/_ping", unix_socket: "/var/run/docker.sock").body
"OK"

Connecting with custom connection options:

iex> Req.get!(url, connect_options: [timeout: 5000])

iex> Req.get!(url, connect_options: [protocol: :http2])

Connecting with built-in CA store (requires OTP 25+):

iex> Req.get!(url, connect_options: [transport_opts: [cacerts: :public_key.cacerts_get()]])

Stream response body using Finch.stream/5:

fun = fn request, finch_request, finch_name, finch_options ->
  fun = fn
    {:status, status}, response ->
      %{response | status: status}

    {:headers, headers}, response ->
      %{response | headers: headers}

    {:data, data}, response ->
      IO.puts(data)
      response
  end

  case Finch.stream(finch_request, finch_name, Req.Response.new(), fun, finch_options) do
    {:ok, response} -> {request, response}
    {:error, exception} -> {request, exception}
  end
end

Req.get!("https://httpbin.org/stream/10", finch_request: fun)

Response Steps

Link to this function

decode_body(request_response)

View Source

Decodes response body based on the detected format.

Supported formats:

FormatDecoder
jsonJason.decode!/2
gzip:zlib.gunzip/1
tar, tgz:erl_tar.extract/2
zip:zip.unzip/2
csvNimbleCSV.RFC4180.parse_string/2 (if nimble_csv is installed)

This step is disabled on response body streaming. If response body is not a binary, in other words it has been transformed by another step, it is left as is.

Request Options

  • :decode_body - if set to false, disables automatic response body decoding. Defaults to true.

  • :decode_json - options to pass to Jason.decode!/2, defaults to [].

  • :raw - if set to true, disables response body decoding. Defaults to false.

Examples

Decode JSON:

iex> response = Req.get!("https://httpbin.org/json")
...> response.body["slideshow"]["title"]
"Sample Slide Show"

Decode gzip:

iex> response = Req.get!("https://httpbin.org/gzip")
...> response.body["gzipped"]
true
Link to this function

decompress_body(request_response)

View Source

Decompresses the response body based on the content-encoding header.

This step is disabled on response body streaming. If response body is not a binary, in other words it has been transformed by another step, it is left as is.

Supported formats:

FormatDecoder
gzip, x-gzip:zlib.gunzip/1
br:brotli.decode/1 (if brotli is installed)
zstd:ezstd.decompress/1 (if ezstd is installed)
otherReturns data as is

This step updates the following headers to reflect the changes:

  • content-length is set to the length of the decompressed body
  • content-encoding is removed

Options

  • :raw - if set to true, disables response body decompression. Defaults to false.

Examples 

iex> response = Req.get!("https://httpbin.org/gzip")
iex> Req.Response.get_header(response, "content-encoding")
["gzip"]
iex> response.body["gzipped"]
true

If the brotli package is installed, Brotli is also supported:

Mix.install([
  :req,
  {:brotli, "~> 0.3.0"}
])

response = Req.get!("https://httpbin.org/brotli")
Req.Response.get_header(response, "content-encoding")
#=> ["br"]
response.body["brotli"]
#=> true
Link to this function

handle_http_errors(request_response)

View Source

Handles HTTP 4xx/5xx error responses.

Request Options

  • :http_errors - how to handle HTTP 4xx/5xx error responses. Can be one of the following:

    • :return (default) - return the response

    • :raise - raise an error

Examples 

iex> Req.get!("https://httpbin.org/status/404").status
404

iex> Req.get!("https://httpbin.org/status/404", http_errors: :raise)
** (RuntimeError) The requested URL returned error: 404
Response body: ""
Link to this function

redirect(request_response)

View Source

Follows redirects.

The original request method may be changed to GET depending on the status code:

CodeMethod handling
301, 302, 303Changed to GET
307, 308Method not changed

Request Options

  • :redirect - if set to false, disables automatic response redirects. Defaults to true.

  • :redirect_trusted - by default, authorization credentials are only sent on redirects with the same host, scheme and port. If :redirect_trusted is set to true, credentials will be sent to any host.

  • :redirect_log_level - the log level to emit redirect logs at. Can also be set to false to disable logging these messsages. Defaults to :debug.

  • :max_redirects - the maximum number of redirects, defaults to 10. If the limit is reached, an error is raised.

Examples 

iex> Req.get!("http://api.github.com").status
# 23:24:11.670 [debug] redirecting to https://api.github.com/
200

iex> Req.get!("https://httpbin.org/redirect/4", max_redirects: 3)
# 23:07:59.570 [debug] redirecting to /relative-redirect/3
# 23:08:00.068 [debug] redirecting to /relative-redirect/2
# 23:08:00.206 [debug] redirecting to /relative-redirect/1
** (RuntimeError) too many redirects (3)

iex> Req.get!("http://api.github.com", redirect_log_level: false)
200

iex> Req.get!("http://api.github.com", redirect_log_level: :error)
# 23:24:11.670 [error]  redirecting to https://api.github.com/
200

Error Steps

Link to this function

retry(request_response_or_error)

View Source

Retries a request in face of errors.

This function can be used as either or both response and error step.

Request Options

  • :retry - can be one of the following:

    • :safe_transient (default) - retry safe (GET/HEAD) requests on HTTP 408/429/500/502/503/504 responses or exceptions with reason field set to :timeout/:econnrefused.

    • :transient - same as :safe_transient except retries all HTTP methods (POST, DELETE, etc.)

    • fun - a 2-arity function that accepts a Req.Request and either a Req.Response or an exception struct and returns one of the following:

      • true - retry with the default delay controller by default delay option described below.

      • {:delay, milliseconds} - retry with the given delay.

      • false/nil - don't retry.

    • false - don't retry.

  • :retry_delay - if not set, which is the default, the retry delay is determined by the value of retry-delay header on HTTP 429/503 responses. If the header is not set, the default delay follows a simple exponential backoff: 1s, 2s, 4s, 8s, ...

    :retry_delay can be set to a function that receives the retry count (starting at 0) and returns the delay, the number of milliseconds to sleep before making another attempt.

  • :retry_log_level - the log level to emit retry logs at. Can also be set to false to disable logging these messsages. Defaults to :error.

  • :max_retries - maximum number of retry attempts, defaults to 3 (for a total of 4 requests to the server, including the initial one.)

Examples

With default options:

iex> Req.get!("https://httpbin.org/status/500,200").status
# 19:02:08.463 [error] retry: got response with status 500, will retry in 2000ms, 2 attempts left
# 19:02:10.710 [error] retry: got response with status 500, will retry in 4000ms, 1 attempt left
200

Delay with jitter:

iex> delay = fn n -> trunc(Integer.pow(2, n) * 1000 * (1 - 0.1 * :rand.uniform())) end
iex> Req.get!("https://httpbin.org/status/500,200", retry_delay: delay).status
# 08:43:19.101 [error] retry: got response with status 500, will retry in 941ms, 2 attempts left
# 08:43:22.958 [error] retry: got response with status 500, will retry in 1877s, 1 attempt left
200