ReqS3 (req_s3 v0.2.3)

ReqS3 handles a custom s3:// url scheme. Example requests are:

# list buckets
Req.get!(req, url: "s3://")

# list objects
Req.get!(req, url: "s3://#{bucket}")

# get object
Req.get!(req, url: "s3://#{bucket}/#{key}")

# put object
Req.put!(req, url: "s3://#{bucket}/#{key}")

The responses for listing buckets and objects are automatically decoded.

Usage

Mix.install([
  {:req, "~> 0.5.0"},
  {:req_s3, "~> 0.2.3"}
])

req = Req.new() |> ReqS3.attach()

Req.get!(req, url: "s3://ossci-datasets").body
#=>
# %{
#   "ListBucketResult" => %{
#     "Contents" => [
#       %{"Key" => "mnist/", ...},
#       %{"Key" => "mnist/t10k-images-idx3-ubyte.gz", ...},
#       ...
#     ],
#     "Name" => "ossci-datasets",
#     ...
#   }
# }

body = Req.get!(req, url: "s3://ossci-datasets/mnist/t10k-images-idx3-ubyte.gz").body
<<_::32, n_images::32, n_rows::32, n_cols::32, _body::binary>> = body
{n_images, n_rows, n_cols}
#=> {10_000, 28, 28}

Examples

Pre-signing

ReqS3 can be used to presign URLs:

options = [
  access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
  secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
]

req = Req.new() |> ReqS3.attach(aws_sigv4: options)
%{status: 200} = Req.put!(req, url: "s3://bucket1/key1", body: "Hello, World!")

presigned_url = ReqS3.presign_url([bucket: "bucket1", key: "key1"] ++ options)
#=> "https://bucket1.s3.amazonaws.com/key1?X-Amz-Algorithm=AWS4-HMAC-SHA256&..."

Req.get!(presigned_url).body
#=> "Hello, World!"

and form uploads:

form = ReqS3.presign_form([bucket: "bucket1", key: "key1"] ++ options)
%{status: 204} = Req.post!(form.url, form_multipart: [file: "Hello, World!"] ++ form.fields)

Req.get!(presigned_url).body
#=> "Hello, World!"

Environment Variables

ReqS3 supports the following standardised system environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
AWS_ENDPOINT_URL_S3

Summary

Functions

attach(request, options \\ [])

Attaches the plugin.

handle_s3_url(request)

Handles the s3:// URL scheme.

presign_form(options)

Returns presigned form for upload.

presign_url(options)

Returns a presigned URL for fetching bucket object contents.

Functions

attach(request, options \\ [])

Attaches the plugin.

handle_s3_url(request)

Handles the s3:// URL scheme.

This request step is automatically added on ReqS3.attach(request).

See module documentation for usage examples.

Request Options

:aws_endpoint_url_s3 - if set, the endpoint URL for S3-compatible services. If AWS_ENDPOINT_URL_S3 system environment variable is set, it is considered first.

presign_form(options)

Returns presigned form for upload.

Options

:access_key_id - the AWS access key id. Defaults to the value of AWS_ACCESS_KEY_ID system environment variable.
:secret_access_key - the AWS secret access key. Defaults to the value of AWS_SECRET_ACCESS_KEY system environment variable.
:region - if set, AWS region. Defaults to the value of AWS_REGION system environment variable, then "us-east-1".
:bucket - the S3 bucket.
:key - the S3 bucket key.
:endpoint_url - if set, the endpoint URL for S3-compatible services. If AWS_ENDPOINT_URL_S3 system environment variable is set, it is considered first.
:content_type - if set, the content-type of the uploaded object.
:max_size - if set, the maximum size of the uploaded object.
:expires_in - the time in milliseconds before the signed upload expires. Defaults to 1h (60 * 60 * 1000 milliseconds).
:datetime - the request datetime, defaults to DateTime.utc_now(:second).

Examples

iex> options = [
...>   access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
...>   secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
...>   bucket: "bucket1",
...>   key: "key1",
...>   expires_in: :timer.hours(1)
...> ]
iex> form = ReqS3.presign_form(options)
iex> form.url
"https://bucket1.s3.amazonaws.com"
iex> form.fields
[
  {"key", "key1"},
  {"policy", "eyJjb25kaXRpb25z...ifQ=="},
  {"x-amz-algorithm", "AWS4-HMAC-SHA256"},
  {"x-amz-credential", "AKIA.../20240528/us-east-1/s3/aws4_request"},
  {"x-amz-date", "20240528T105226Z"},
  {"x-amz-server-side-encryption", "AES256"},
  {"x-amz-signature", "465315d202fbb2ce081f79fca755a958a18ff68d253e6d2a611ca4b2292d8925"}
]

presign_url(options)

Returns a presigned URL for fetching bucket object contents.

Options

:access_key_id - the AWS access key id. Defaults to the value of AWS_ACCESS_KEY_ID system environment variable.
:secret_access_key - the AWS secret access key. Defaults to the value of AWS_SECRET_ACCESS_KEY system environment variable.
:region - the AWS region. Defaults to the value of AWS_REGION system environment variable, then "us-east-1".
:url - the URL to presign, for example: "https://{bucket}.s3.amazonaws.com/{key}", s3://{bucket}/{key}, etc. s3:// URL uses :endpoint_url option described below.
Instead of passing the :url option, you can instead pass :bucket and :key options which will generate https://{bucket}.s3.amazonaws.com/{key} (or {endpoint_url}/{bucket}/{key}).
:endpoint_url - if set, the endpoint URL for S3-compatible services. If AWS_ENDPOINT_URL_S3 system environment variable is set, it is considered first.

Examples

Note: This example assumes AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set.

iex> req = Req.new() |> ReqS3.attach()
iex> bucket = System.fetch_env!("BUCKET_NAME")
iex> key = "key1"
iex> %{status: 200} = Req.put!(req, url: "s3://#{bucket}/#{key}", body: "Hello, World!")
iex> url = ReqS3.presign_url(bucket: bucket, key: key)
iex> url =~ "https://#{bucket}.s3.amazonaws.com/#{key}?X-Amz-Algorithm=AWS4-HMAC-SHA256&"
true
iex> %{status: 200, body: body} = Req.get!(url)
iex> body
"Hello, World!"