ReqS3 (req_s3 v0.2.0)

Req plugin for Amazon S3 and S3 API-compatible services.

ReqS3 handles a custom s3:// url scheme that supports two endpoints:

s3://<bucket>        # list bucket items
s3://<bucket>/<item> # get item content

Examples

iex> req = Req.new() |> ReqS3.attach()
iex> Req.get!(req, url: "s3://ossci-datasets").body
%{
  "ListBucketResult" => %{
    "Contents" => [
      %{"Key" => "mnist/", ...},
      %{"Key" => "mnist/t10k-images-idx3-ubyte.gz", ...},
      ...
    ],
    "Name" => "ossci-datasets",
    ...
  }
}

iex> req = Req.new() |> ReqS3.attach()
iex> body = Req.get!(req, url: "s3://ossci-datasets/mnist/t10k-images-idx3-ubyte.gz").body
iex> <<_::32, n_images::32, n_rows::32, n_cols::32, _body::binary>> = body
iex> {n_images, n_rows, n_cols}
{10_000, 28, 28}

Pre-signing

See presign_url/1 and presign_form/1.

Summary

Functions

attach(request, options \\ [])

Attaches the plugin.

presign_form(options)

Returns presigned form for upload.

presign_url(options)

Returns a presigned URL for fetching bucket object contents.

Functions

attach(request, options \\ [])

Attaches the plugin.

presign_form(options)

Returns presigned form for upload.

Options

:access_key_id - the access key id.
:secret_access_key - the secret access key.
:region - the S3 region, defaults to "us-east-1".
:bucket - the S3 bucket.
:key - the S3 bucket key.
:content_type - if set, the content-type of the uploaded object.
:max_size - if set, the maximum size of the uploaded object.
:expires_in - the time in milliseconds before the signed upload expires. Defaults to 1h (60 * 60 * 1000 milliseconds).
:datetime - the request datetime, defaults to DateTime.utc_now(:second).

Examples

iex> options = [
...>   access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
...>   secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
...>   bucket: "bucket1",
...>   key: "key1",
...>   expires_in: :timer.hours(1)
...> ]
iex> %{url: url, fields: fields} = ReqS3.presign_form(options)
iex> url
"https://bucket1.s3.amazonaws.com"
iex> fields
%{
  "key" => "key1",
  "policy" => "eyJjb25kaXRpb25z...ifQ==",
  "x-amz-algorithm" => "AWS4-HMAC-SHA256",
  "x-amz-credential" => "AKIA.../20240528/us-east-1/s3/aws4_request",
  "x-amz-date" => "20240528T105226Z",
  "x-amz-server-side-encryption" => "AES256",
  "x-amz-signature" => "465315d202fbb2ce081f79fca755a958a18ff68d253e6d2a611ca4b2292d8925"
}

presign_url(options)

Returns a presigned URL for fetching bucket object contents.

Options

:access_key_id - the AWS access key id.
:secret_access_key - the AWS secret access key.
:region - if set, AWS region. Defaults to "us-east-1".
:url - the URL to presign, for example: "https://bucket.s3.amazonaws.com" or s3://bucket.
Instead of passing the :url option, you can also pass :bucket and :key options which will generate a https://{bucket}.s3.amazonaws.com/{key} url.

Examples

iex> options = [
...>   access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
...>   secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
...> ]
iex> req = Req.new() |> ReqS3.attach(aws_sigv4: options)
iex> %{status: 200} = Req.put!(req, url: "s3://wojtekmach-test/key1", body: "Hello, World!")
iex> url = ReqS3.presign_url([url: "s3://wojtekmach-test/key1"] ++ options)
iex> "https://wojtekmach-test.s3.amazonaws.com/key1?X-Amz-Algorithm=AWS4-HMAC" <> _ = url
iex> %{status: 200, body: body} = Req.get!(url)
iex> body
"Hello, World!"