req
v0.3.10

View Source Req.Steps (req v0.3.10)

The collection of built-in steps.

Req is composed of three main pieces:

  • Req - the high-level API

  • Req.Request - the low-level API and the request struct

  • Req.Steps - the collection of built-in steps (you're here!)

Link to this section Summary

Request steps

auth(request)

Sets request authentication.

cache(request)

Performs HTTP caching using if-modified-since header.

compress_body(request)

Compresses the request body.

compressed(request)

Asks the server to return compressed response.

encode_body(request)

Encodes the request body.

put_base_url(request)

Sets base URL for all requests.

put_params(request)

Adds params to request query string.

put_path_params(request)

Uses a templated request path.

put_plug(request)

Runs the request against a plug instead of over the network.

put_range(request)

Sets the "Range" request header.

put_user_agent(request)

Sets the user-agent header.

run_finch(request)

Runs the request using Finch.

Response steps

decode_body(request_response)

Decodes response body based on the detected format.

decompress_body(request_response)

Decompresses the response body based on the content-encoding header.

follow_redirects(request_response)

Follows redirects.

handle_http_errors(request_response)

Handles HTTP 4xx/5xx error responses.

output(request_response)

Writes the response body to a file.

Error steps

retry(request_response_or_error)

Retries a request in face of errors.

Link to this section Request steps

Link to this function

auth(request)

View Source

Sets request authentication.

request-options

 Request Options

  • :auth - sets the authorization header:

    • string - sets to this value;

    • {username, password} - uses Basic HTTP authentication;

    • {:bearer, token} - uses Bearer HTTP authentication;

    • :netrc - load credentials from .netrc at path specified in NETRC environment variable. If NETRC is not set, load .netrc in user's home directory;

    • {:netrc, path} - load credentials from path

examples

 Examples 

iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {"bad", "bad"}).status
401
iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {"foo", "bar"}).status
200

iex> Req.get!("https://httpbin.org/bearer", auth: {:bearer, ""}).status
401
iex> Req.get!("https://httpbin.org/bearer", auth: {:bearer, "foo"}).status
200

iex> System.put_env("NETRC", "./test/my_netrc")
iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: :netrc).status
200

iex> Req.get!("https://httpbin.org/basic-auth/foo/bar", auth: {:netrc, "./test/my_netrc"}).status
200
Link to this function

cache(request)

View Source

Performs HTTP caching using if-modified-since header.

Only successful (200 OK) responses are cached.

This step also prepends a response step that loads and writes the cache. Be careful when prepending other response steps, make sure the cache is loaded/written as soon as possible.

options

 Options

  • :cache - if true, performs caching. Defaults to false.

  • :cache_dir - the directory to store the cache, defaults to <user_cache_dir>/req (see: :filename.basedir/3)

examples

 Examples 

iex> url = "https://elixir-lang.org"
iex> response1 = Req.get!(url, cache: true)
iex> response2 = Req.get!(url, cache: true)
iex> response1 == response2
true
Link to this function

compress_body(request)

View Source

Compresses the request body.

request-options

 Request Options

  • :compress_body - if set to true, compresses the request body using gzip. Defaults to false.
Link to this function

compressed(request)

View Source

Asks the server to return compressed response.

Supported formats:

  • gzip
  • deflate
  • br (if brotli is installed)
  • zstd (if ezstd is installed)

request-options

 Request Options

  • :compressed - if set to true, sets the accept-encoding header with compression algorithms that Req supports. Defaults to true.

examples

 Examples

Req automatically decompresses response body (decompress_body/1 step) so let's disable that by passing raw: true.

By default, we ask the server to send compressed response. Let's look at the headers and the raw body. Notice the body starts with <<31, 139>> (<<0x1F, 0x8B>>), the "magic bytes" for gzip:

iex> response = Req.get!("https://elixir-lang.org", raw: true)
iex> response.headers |> List.keyfind("content-encoding", 0)
{"content-encoding", "gzip"}
iex> response.body |> binary_part(0, 2)
<<31, 139>>

Now, let's pass compressed: false and notice the raw body was not compressed:

iex> response = Req.get!("https://elixir-lang.org", raw: true, compressed: false)
iex> response.headers |> List.keyfind("content-encoding", 0)
nil
iex> response.body |> binary_part(0, 15)
"<!DOCTYPE html>"

The Brotli and Zstandard compression algorithms are also supported if the optional packages are installed:

Mix.install([
  :req,
  {:brotli, "~> 0.3.0"},
  {:ezstd, "~> 1.0"}
])

response = Req.get!("https://httpbin.org/anything")
response.body["headers"]["Accept-Encoding"]
#=> "zstd, br, gzip, deflate"
Link to this function

encode_body(request)

View Source

Encodes the request body.

request-options

 Request Options

  • :form - if set, encodes the request body as form data (using URI.encode_query/1).

  • :json - if set, encodes the request body as JSON (using Jason.encode_to_iodata!/1), sets the accept header to application/json, and the content-type header to application/json.

examples

 Examples 

iex> Req.post!("https://httpbin.org/anything", form: [x: 1]).body["form"]
%{"x" => "1"}

iex> Req.post!("https://httpbin.org/post", json: %{x: 2}).body["json"]
%{"x" => 2}
Link to this function

put_base_url(request)

View Source

Sets base URL for all requests.

request-options

 Request Options

  • :base_url - if set, the request URL is merged with this base URL.

examples

 Examples 

iex> req = Req.new(base_url: "https://httpbin.org")
iex> Req.get!(req, url: "/status/200").status
200
iex> Req.get!(req, url: "/status/201").status
201
Link to this function

put_params(request)

View Source

Adds params to request query string.

request-options

 Request Options

  • :params - params to add to the request query string. Defaults to [].

examples

 Examples 

iex> Req.get!("https://httpbin.org/anything/query", params: [x: 1, y: 2]).body["args"]
%{"x" => "1", "y" => "2"}
Link to this function

put_path_params(request)

View Source

Uses a templated request path.

request-options

 Request Options

  • :path_params - params to add to the templated path. Defaults to [].

examples

 Examples 

iex> Req.get!("https://httpbin.org/status/:code", path_params: [code: 200]).status
200
Link to this function

put_plug(request)

View Source

Runs the request against a plug instead of over the network.

request-options

 Request Options

  • :plug - if set, the plug to run the request against.

examples

 Examples

This step is particularly useful to test plugs:

defmodule Echo do
  def call(conn, _) do
    "/" <> path = conn.request_path
    Plug.Conn.send_resp(conn, 200, path)
  end
end

test "echo" do
  assert Req.get!("http:///hello", plug: Echo).body == "hello"
end

You can define plugs as functions too:

test "echo" do
  echo = fn conn ->
    "/" <> path = conn.request_path
    Plug.Conn.send_resp(conn, 200, path)
  end

  assert Req.get!("http:///hello", plug: echo).body == "hello"
end

which is particularly useful to create HTTP service mocks with tools like Bypass.

Here is another example, let's run the request against Plug.Static pointed to the Req's source code and fetch the README:

iex> resp = Req.get!("http:///README.md", plug: {Plug.Static, at: "/", from: "."})
iex> resp.body =~ "Req is a batteries-included HTTP client for Elixir."
true
Link to this function

put_range(request)

View Source

Sets the "Range" request header.

request-options

 Request Options

  • :range - can be one of the following:

    • a string - returned as is

    • a first..last range - converted to "bytes=<first>-<last>"

examples

 Examples 

iex> response = Req.get!("https://httpbin.org/range/100", range: 0..3)
iex> response.status
206
iex> response.body
"abcd"
iex> List.keyfind(response.headers, "content-range", 0)
{"content-range", "bytes 0-3/100"}
Link to this function

put_user_agent(request)

View Source

Sets the user-agent header.

request-options

 Request Options

  • :user_agent - sets the user-agent header. Defaults to "req/0.3.10".

examples

 Examples 

iex> Req.get!("https://httpbin.org/user-agent").body
%{"user-agent" => "req/0.3.10"}

iex> Req.get!("https://httpbin.org/user-agent", user_agent: "foo").body
%{"user-agent" => "foo"}
Link to this function

run_finch(request)

View Source

Runs the request using Finch.

This is the default Req adapter. See "Adapter" section in the Req.Request module documentation for more information on adapters.

request-options

 Request Options

  • :finch - the name of the Finch pool. Defaults to a pool automatically started by Req. The default pool uses HTTP/1 although that may change in the future.

  • :connect_options - dynamically starts (or re-uses already started) Finch pool with the given connection options:

    • :timeout - socket connect timeout in milliseconds, defaults to 30_000.

    • :protocol - the HTTP protocol to use, defaults to :http1.

    • :hostname - Mint explicit hostname, see Mint.HTTP.connect/4 for more information.

    • :transport_opts - Mint transport options, see Mint.HTTP.connect/4 for more information.

    • :proxy_headers - Mint proxy headers, see Mint.HTTP.connect/4 for more information.

    • :proxy - Mint HTTP/1 proxy settings, a {schema, address, port, options} tuple. See Mint.HTTP.connect/4 for more information.

    • :client_settings - Mint HTTP/2 client settings, see Mint.HTTP.connect/4 for more information.

  • :pool_timeout - pool checkout timeout in milliseconds, defaults to 5000.

  • :receive_timeout - socket receive timeout in milliseconds, defaults to 15_000.

  • :unix_socket - if set, connect through the given UNIX domain socket

  • :finch_request - a function that executes the Finch request, defaults to using Finch.request/3.

    The function should accept 4 arguments:

    • request - the %Req.Request{} struct

    • finch_request - the Finch request

    • finch_name - the Finch name

    • finch_options - the Finch options

    And it should return either {request, response} or {request, exception}.

examples

 Examples

Custom :receive_timeout:

iex> Req.get!(url, receive_timeout: 1000)

Connecting through UNIX socket:

iex> Req.get!("http:///v1.41/_ping", unix_socket: "/var/run/docker.sock").body
"OK"

Connecting with custom connection options:

iex> Req.get!(url, connect_options: [timeout: 5000])

iex> Req.get!(url, connect_options: [protocol: :http2])

Connecting with built-in CA store (requires OTP 25+):

iex> Req.get!(url, connect_options: [transport_opts: [cacerts: :public_key.cacerts_get()]])

Stream response body:

fun = fn request, finch_request, finch_name, finch_options ->
  fun = fn
    {:status, status}, response ->
      %{response | status: status}

    {:headers, headers}, response ->
      %{response | headers: headers}

    {:data, data}, response ->
      IO.puts(data)
      response
  end

  case Finch.stream(finch_request, finch_name, Req.Response.new(), fun, finch_options) do
    {:ok, response} -> {request, response}
    {:error, exception} -> {request, exception}
  end
end

Req.get!("https://httpbin.org/stream/10", finch_request: fun)

Link to this section Response steps

Link to this function

decode_body(request_response)

View Source

Decodes response body based on the detected format.

Supported formats:

FormatDecoder
jsonJason.decode!/2
gzip:zlib.gunzip/1
tar, tgz:erl_tar.extract/2
zip:zip.unzip/2
csvNimbleCSV.RFC4180.parse_string/2 (if nimble_csv is installed)

request-options

 Request Options

  • :decode_body - if set to false, disables automatic response body decoding. Defaults to true.

  • :decode_json - options to pass to Jason.decode!/2, defaults to [].

  • :raw - if set to true, disables response body decoding. Defaults to false.

examples

 Examples

Decode JSON:

iex> response = Req.get!("https://httpbin.org/json")
...> response.body["slideshow"]["title"]
"Sample Slide Show"

Decode gzip:

iex> response = Req.get!("https://httpbin.org/gzip")
...> response.body["gzipped"]
true
Link to this function

decompress_body(request_response)

View Source

Decompresses the response body based on the content-encoding header.

Supported formats:

FormatDecoder
gzip, x-gzip:zlib.gunzip/1
deflate:zlib.unzip/1
br:brotli.decode/1 (if brotli is installed)
zstd:ezstd.decompress/1 (if ezstd is installed)
identityReturns data as is

options

 Options

  • :raw - if set to true, disables response body decompression. Defaults to false.

examples

 Examples 

iex> response = Req.get!("https://httpbin.org/gzip")
iex> List.keyfind(response.headers, "content-encoding", 0)
{"content-encoding", "gzip"}
iex> response.body["gzipped"]
true

If the brotli package is installed, Brotli is also supported:

Mix.install([
  :req,
  {:brotli, "~> 0.3.0"}
])

response = Req.get!("https://httpbin.org/brotli")
response.headers |> List.keyfind("content-encoding", 0)
#=> {"content-encoding", "br"}
response.body["brotli"]
#=> true
Link to this function

follow_redirects(request_response)

View Source

Follows redirects.

The original request method may be changed to GET depending on the status code:

CodeMethod handling
301, 302, 303Changed to GET
307, 308Method not changed

request-options

 Request Options

  • :follow_redirects - if set to false, disables automatic response redirects. Defaults to true.

  • :location_trusted - by default, authorization credentials are only sent on redirects with the same host, scheme and port. If :location_trusted is set to true, credentials will be sent to any host.

  • :max_redirects - the maximum number of redirects, defaults to 10. If the limit is reached, an error is raised.

  • :redirect_log_level - the log level to emit redirect logs at. Can also be set to false to disable logging these messsages. Defaults to :debug.

examples

 Examples 

iex> Req.get!("http://api.github.com").status
# 23:24:11.670 [debug]  follow_redirects: redirecting to https://api.github.com/
200

iex> Req.get!("https://httpbin.org/redirect/4", max_redirects: 3)
# 23:07:59.570 [debug] follow_redirects: redirecting to /relative-redirect/3
# 23:08:00.068 [debug] follow_redirects: redirecting to /relative-redirect/2
# 23:08:00.206 [debug] follow_redirects: redirecting to /relative-redirect/1
** (RuntimeError) too many redirects (3)

iex> Req.get!("http://api.github.com", redirect_log_level: false)
200

iex> Req.get!("http://api.github.com", redirect_log_level: :error)
# 23:24:11.670 [error]  follow_redirects: redirecting to https://api.github.com/
200
Link to this function

handle_http_errors(request_response)

View Source

Handles HTTP 4xx/5xx error responses.

request-options

 Request Options

  • :http_errors - how to handle HTTP 4xx/5xx error responses. Can be one of the following:

    • :return (default) - return the response

    • :raise - raise an error

examples

 Examples 

iex> Req.get!("https://httpbin.org/status/404").status
404

iex> Req.get!("https://httpbin.org/status/404", http_errors: :raise)
** (RuntimeError) The requested URL returned error: 404
Response body: ""
Link to this function

output(request_response)

View Source

Writes the response body to a file.

After the output file is written, the response body is set to "".

request-options

 Request Options

  • :output - if set, writes the response body to a file. Can be one of:

    • path - writes to the given path

    • :remote_name - uses the remote name as the filename in the current working directory

examples

 Examples 

iex> Req.get!("https://elixir-lang.org/index.html", output: "/tmp/elixir_home.html")
iex> File.exists?("/tmp/elixir_home.html")
true

iex> Req.get!("https://elixir-lang.org/blog/index.html", output: :remote_name)
iex> File.exists?("index.html")
true

Link to this section Error steps

Link to this function

retry(request_response_or_error)

View Source

Retries a request in face of errors.

This function can be used as either or both response and error step.

request-options

 Request Options

  • :retry - can be one of the following:

    • :safe (default) - retry GET/HEAD requests on HTTP 408/429/5xx responses or exceptions

    • fun - a 1-arity function that accepts either a Req.Response or an exception struct and returns boolean whether to retry

    • false - never retry

  • :retry_delay - a function that receives the retry count (starting at 0) and returns the delay, the number of milliseconds to sleep before making another attempt. Defaults to a simple exponential backoff: 1s, 2s, 4s, 8s, ...

    If the response is HTTP 429 and contains the retry-after header, the value of the header is used to determine the next retry delay.

  • :retry_log_level - the log level to emit retry logs at. Can also be set to false to disable logging these messsages. Defaults to :error.

  • :max_retries - maximum number of retry attempts, defaults to 3 (for a total of 4 requests to the server, including the initial one.)

examples

 Examples

With default options:

iex> Req.get!("https://httpbin.org/status/500,200").status
# 19:02:08.463 [error] retry: got response with status 500, will retry in 2000ms, 2 attempts left
# 19:02:10.710 [error] retry: got response with status 500, will retry in 4000ms, 1 attempt left
200

Delay with jitter:

iex> delay = fn n -> trunc(Integer.pow(2, n) * 1000 * (1 - 0.1 * :rand.uniform())) end
iex> Req.get!("https://httpbin.org/status/500,200", retry_delay: delay).status
# 08:43:19.101 [error] retry: got response with status 500, will retry in 941ms, 2 attempts left
# 08:43:22.958 [error] retry: got response with status 500, will retry in 1877s, 1 attempt left
200