ExAws.S3 v2.0.2 ExAws.S3 View Source

Operations on AWS S3

Basic Operations

The vast majority of operations here represent a single operation on S3.

Examples

S3.list_objects |> ExAws.request! #=> %{body: [list, of, objects]}
S3.list_objects |> ExAws.stream! |> Enum.to_list #=> [list, of, objects]

S3.put_object("my-bucket", "path/to/bucket", contents) |> ExAws.request!

Higher Level Operations

There are also some operations which operate at a higher level to make it easier to download and upload very large files.

Multipart uploads

"path/to/big/file"
|> S3.Upload.stream_file
|> S3.upload("my-bucket", "path/on/s3")
|> ExAws.request #=> {:ok, :done}

Download large file to disk

S3.download_file("my-bucket", "path/on/s3", "path/to/dest/file")
|> ExAws.request #=> {:ok, :done}

More high level functionality

Task.async_stream makes some high level flows so easy you don't need explicit ExAws support.

For example, here is how to concurrently upload many files.

upload_file = fn {src_path, dest_path} ->
  S3.put_object("my_bucket", dest_path, File.read!(src_path))
  |> ExAws.request!
end

paths = %{"path/to/src0" => "path/to/dest0", "path/to/src1" => "path/to/dest1"}

paths
|> Task.async_stream(upload_file, max_concurrency: 10)
|> Stream.run

Link to this section Summary

Functions

Delete a bucket

Delete a bucket cors

Delete a bucket lifecycle

Delete a bucket policy

Delete a bucket replication

Delete a bucket tagging

Delete a bucket website

Delete multiple objects within a bucket

Delete object object in bucket

Remove the entire tag set from the specified object

Download an S3 Object to a file.

Get bucket acl

Get bucket cors

Get bucket lifecycle

Get bucket location

Get bucket logging

Get bucket notification

Get bucket policy

Get bucket replication

Get bucket payment configuration

Get bucket tagging

Get bucket versioning

Get bucket website

Get an object from a bucket

Get an object's access control policy

Get a torrent for a bucket

Determine if a bucket exists

Determine of an object exists

List multipart uploads for a bucket

List objects in bucket

List the parts of a multipart upload

Restore an object to a particular version

Generates a pre-signed URL for this object.

Creates a bucket in the specified region

Update or create a bucket bucket access control

Update or create a bucket CORS policy

Update or create a bucket lifecycle configuration

Update or create a bucket logging configuration

Update or create a bucket notification configuration

Update or create a bucket policy configuration

Update or create a bucket replication configuration

Update or create a bucket requestPayment configuration

Update or create a bucket tagging configuration

Update or create a bucket versioning configuration

Update or create a bucket website configuration

Create an object within a bucket

Create or update an object's access control FIXME

Add a set of tags to an existing object

Link to this section Types

Link to this type

acl_opts() View Source
acl_opts() :: {:acl, canned_acl()} | grant()

Link to this type

amz_meta_opts() View Source
amz_meta_opts() :: [{atom(), binary()} | {binary(), binary()}, ...]

Link to this type

canned_acl() View Source
canned_acl() ::
  :private
  | :public_read
  | :public_read_write
  | :authenticated_read
  | :bucket_owner_read
  | :bucket_owner_full_control

Link to this type

customer_encryption_opts() View Source
customer_encryption_opts() :: [
  customer_algorithm: binary(),
  customer_key: binary(),
  customer_key_md5: binary()
]

Link to this type

download_file_opts() View Source
download_file_opts() :: [
  max_concurrency: pos_integer(),
  chunk_size: pos_integer(),
  timeout: pos_integer()
]

Link to this type

encryption_opts() View Source
encryption_opts() ::
  binary() | [{:aws_kms_key_id, binary()}] | customer_encryption_opts()

Link to this type

get_object_opts() View Source
get_object_opts() :: [
  {:response, get_object_response_opts()} | head_object_opt()
]

Link to this type

get_object_response_opts() View Source
get_object_response_opts() :: [
  content_language: binary(),
  expires: binary(),
  cache_control: binary(),
  content_disposition: binary(),
  content_encoding: binary()
]

Link to this type

grant() View Source
grant() ::
  {:grant_read, grantee()}
  | {:grant_read_acp, grantee()}
  | {:grant_write_acp, grantee()}
  | {:grant_full_control, grantee()}

Link to this type

grantee() View Source
grantee() :: [email: binary(), id: binary(), uri: binary()]

Link to this type

head_object_opt() View Source
head_object_opt() ::
  {:encryption, customer_encryption_opts()}
  | {:range, binary()}
  | {:if_modified_since, binary()}
  | {:if_unmodified_since, binary()}
  | {:if_match, binary()}
  | {:if_none_match, binary()}

Link to this type

head_object_opts() View Source
head_object_opts() :: [head_object_opt()]

Link to this type

initiate_multipart_upload_opts() View Source
initiate_multipart_upload_opts() :: [
  {:cache_control, binary()}
  | {:content_disposition, binary()}
  | {:content_encoding, binary()}
  | {:content_type, binary()}
  | {:expires, binary()}
  | {:storage_class, :standard | :redunced_redundancy}
  | {:website_redirect_location, binary()}
  | {:encryption, encryption_opts()}
  | acl_opts()
]

Link to this type

list_objects_opts() View Source
list_objects_opts() :: [
  delimiter: binary(),
  marker: binary(),
  prefix: binary(),
  encoding_type: binary(),
  max_keys: 0..1000,
  stream_prefixes: boolean()
]

Link to this type

presigned_url_opts() View Source
presigned_url_opts() :: [
  expires_in: integer(),
  virtual_host: boolean(),
  query_params: [{binary(), binary()}]
]

Link to this type

pub_object_copy_opts() View Source
pub_object_copy_opts() :: [
  {:metadata_directive, :COPY | :REPLACE}
  | {:copy_source_if_modified_since, binary()}
  | {:copy_source_if_unmodified_since, binary()}
  | {:copy_source_if_match, binary()}
  | {:copy_source_if_none_match, binary()}
  | {:website_redirect_location, binary()}
  | {:destination_encryption, encryption_opts()}
  | {:source_encryption, customer_encryption_opts()}
  | {:cache_control, binary()}
  | {:content_disposition, binary()}
  | {:content_encoding, binary()}
  | {:content_length, binary()}
  | {:content_type, binary()}
  | {:expect, binary()}
  | {:expires, binary()}
  | {:storage_class, :standard | :redunced_redundancy}
  | {:website_redirect_location, binary()}
  | {:meta, amz_meta_opts()}
  | acl_opts()
]

Link to this type

put_object_opts() View Source
put_object_opts() :: [
  {:cache_control, binary()}
  | {:content_disposition, binary()}
  | {:content_encoding, binary()}
  | {:content_length, binary()}
  | {:content_type, binary()}
  | {:expect, binary()}
  | {:expires, binary()}
  | {:storage_class, :standard | :redunced_redundancy}
  | {:website_redirect_location, binary()}
  | {:encryption, encryption_opts()}
  | {:meta, amz_meta_opts()}
  | acl_opts()
]

Link to this type

upload_opts() View Source
upload_opts() :: [
  {:max_concurrency, pos_integer()} | initiate_multipart_upload_opts()
]

Link to this type

upload_part_copy_opts() View Source
upload_part_copy_opts() :: [
  copy_source_range: Range.t(),
  copy_source_if_modified_since: binary(),
  copy_source_if_unmodified_since: binary(),
  copy_source_if_match: binary(),
  copy_source_if_none_match: binary(),
  destination_encryption: encryption_opts(),
  source_encryption: customer_encryption_opts()
]

Link to this section Functions

Link to this function

abort_multipart_upload(bucket, object, upload_id) View Source
abort_multipart_upload(
  bucket :: binary(),
  object :: binary(),
  upload_id :: binary()
) :: ExAws.Operation.S3.t()

Abort a multipart upload

Link to this function

complete_multipart_upload(bucket, object, upload_id, parts) View Source
complete_multipart_upload(
  bucket :: binary(),
  object :: binary(),
  upload_id :: binary(),
  parts :: [{binary() | pos_integer(), binary()}, ...]
) :: ExAws.Operation.S3.t()

Complete a multipart upload

Link to this function

delete_all_objects(bucket, objects, opts \\ []) View Source
delete_all_objects(
  bucket :: binary(),
  objects :: [binary() | {binary(), binary()}, ...] | Enumerable.t(),
  opts :: [{:quiet, true}]
) :: ExAws.Operation.S3DeleteAllObjects.t()

Delete all listed objects.

When performed, this function will continue making delete_multiple_objects requests deleting 1000 objects at a time until all are deleted.

Can be streamed.

Example

stream = ExAws.S3.list_objects(bucket(), prefix: "some/prefix") |> ExAws.stream!() |> Stream.map(& &1.key)
ExAws.S3.delete_all_objects(bucket(), stream) |> ExAws.request() 
Link to this function

delete_bucket(bucket) View Source
delete_bucket(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket

Link to this function

delete_bucket_cors(bucket) View Source
delete_bucket_cors(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket cors

Link to this function

delete_bucket_lifecycle(bucket) View Source
delete_bucket_lifecycle(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket lifecycle

Link to this function

delete_bucket_policy(bucket) View Source
delete_bucket_policy(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket policy

Link to this function

delete_bucket_replication(bucket) View Source
delete_bucket_replication(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket replication

Link to this function

delete_bucket_tagging(bucket) View Source
delete_bucket_tagging(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket tagging

Link to this function

delete_bucket_website(bucket) View Source
delete_bucket_website(bucket :: binary()) :: ExAws.Operation.S3.t()

Delete a bucket website

Link to this function

delete_multiple_objects(bucket, objects, opts \\ []) View Source
delete_multiple_objects(
  bucket :: binary(),
  objects :: [binary() | {binary(), binary()}, ...],
  opts :: [{:quiet, true}]
) :: ExAws.Operation.S3.t()

Delete multiple objects within a bucket

Limited to 1000 objects.

Link to this function

delete_object(bucket, object, opts \\ []) View Source

Delete object object in bucket

Link to this function

delete_object_tagging(bucket, object, opts \\ []) View Source
delete_object_tagging(
  bucket :: binary(),
  object :: binary(),
  opts :: Keyword.t()
) :: ExAws.Operation.S3.t()

Remove the entire tag set from the specified object

Link to this function

download_file(bucket, path, dest, opts \\ []) View Source
download_file(
  bucket :: binary(),
  path :: binary(),
  dest :: binary(),
  opts :: download_file_opts()
) :: ExAws.S3.Download.t()

Download an S3 Object to a file.

This operation download multiple parts of an S3 object concurrently, allowing you to maximize throughput.

Defaults to a concurrency of 8, chunk size of 1MB, and a timeout of 1 minute.

Link to this function

get_bucket_acl(bucket) View Source
get_bucket_acl(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket acl

Link to this function

get_bucket_cors(bucket) View Source
get_bucket_cors(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket cors

Link to this function

get_bucket_lifecycle(bucket) View Source
get_bucket_lifecycle(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket lifecycle

Link to this function

get_bucket_location(bucket) View Source
get_bucket_location(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket location

Link to this function

get_bucket_logging(bucket) View Source
get_bucket_logging(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket logging

Link to this function

get_bucket_notification(bucket) View Source
get_bucket_notification(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket notification

Link to this function

get_bucket_object_versions(bucket, opts \\ []) View Source
get_bucket_object_versions(bucket :: binary(), opts :: Keyword.t()) ::
  ExAws.Operation.S3.t()

Get bucket object versions

Link to this function

get_bucket_policy(bucket) View Source
get_bucket_policy(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket policy

Link to this function

get_bucket_replication(bucket) View Source
get_bucket_replication(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket replication

Link to this function

get_bucket_request_payment(bucket) View Source
get_bucket_request_payment(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket payment configuration

Link to this function

get_bucket_tagging(bucket) View Source
get_bucket_tagging(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket tagging

Link to this function

get_bucket_versioning(bucket) View Source
get_bucket_versioning(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket versioning

Link to this function

get_bucket_website(bucket) View Source
get_bucket_website(bucket :: binary()) :: ExAws.Operation.S3.t()

Get bucket website

Link to this function

get_object(bucket, object, opts \\ []) View Source
get_object(bucket :: binary(), object :: binary(), opts :: get_object_opts()) ::
  ExAws.Operation.S3.t()

Get an object from a bucket

Link to this function

get_object_acl(bucket, object, opts \\ []) View Source
get_object_acl(bucket :: binary(), object :: binary(), opts :: Keyword.t()) ::
  ExAws.Operation.S3.t()

Get an object's access control policy

Link to this function

get_object_tagging(bucket, object, opts \\ []) View Source
get_object_tagging(bucket :: binary(), object :: binary(), opts :: Keyword.t()) ::
  ExAws.Operation.S3.t()

Get object tagging

Link to this function

get_object_torrent(bucket, object) View Source
get_object_torrent(bucket :: binary(), object :: binary()) ::
  ExAws.Operation.S3.t()

Get a torrent for a bucket

Link to this function

head_bucket(bucket) View Source
head_bucket(bucket :: binary()) :: ExAws.Operation.S3.t()

Determine if a bucket exists

Link to this function

head_object(bucket, object, opts \\ []) View Source
head_object(bucket :: binary(), object :: binary(), opts :: head_object_opts()) ::
  ExAws.Operation.S3.t()

Determine of an object exists

Link to this function

initiate_multipart_upload(bucket, object, opts \\ []) View Source
initiate_multipart_upload(
  bucket :: binary(),
  object :: binary(),
  opts :: initiate_multipart_upload_opts()
) :: ExAws.Operation.S3.t()

Initiate a multipart upload

Link to this function

list_buckets(opts \\ []) View Source
list_buckets(opts :: Keyword.t()) :: ExAws.Operation.S3.t()

List buckets

Link to this function

list_multipart_uploads(bucket, opts \\ []) View Source
list_multipart_uploads(bucket :: binary(), opts :: Keyword.t()) ::
  ExAws.Operation.S3.t()

List multipart uploads for a bucket

Link to this function

list_objects(bucket, opts \\ []) View Source
list_objects(bucket :: binary(), opts :: list_objects_opts()) ::
  ExAws.Operation.S3.t()

List objects in bucket

Can be streamed.

Examples

S3.list_objects("my-bucket") |> ExAws.request

S3.list_objects("my-bucket") |> ExAws.stream!
S3.list_objects("my-bucket", delimiter: "/", prefix: "backup") |> ExAws.stream!
S3.list_objects("my-bucket", prefix: "some/inner/location/path") |> ExAws.stream!
S3.list_objects("my-bucket", max_keys: 5, encoding_type: "url") |> ExAws.stream!
Link to this function

list_parts(bucket, object, upload_id, opts \\ []) View Source
list_parts(
  bucket :: binary(),
  object :: binary(),
  upload_id :: binary(),
  opts :: Keyword.t()
) :: ExAws.Operation.S3.t()

List the parts of a multipart upload

Link to this function

options_object(bucket, object, origin, request_method, request_headers \\ []) View Source
options_object(
  bucket :: binary(),
  object :: binary(),
  origin :: binary(),
  request_method :: atom(),
  request_headers :: [binary()]
) :: ExAws.Operation.S3.t()

Determine the CORS configuration for an object

Link to this function

post_object_restore(bucket, object, number_of_days, opts \\ []) View Source
post_object_restore(
  bucket :: binary(),
  object :: binary(),
  number_of_days :: pos_integer(),
  opts :: [{:version_id, binary()}]
) :: ExAws.Operation.S3.t()

Restore an object to a particular version

Link to this function

presigned_url(config, http_method, bucket, object, opts \\ []) View Source
presigned_url(
  config :: map(),
  http_method :: atom(),
  bucket :: binary(),
  object :: binary(),
  opts :: presigned_url_opts()
) :: {:ok, binary()} | {:error, binary()}

Generates a pre-signed URL for this object.

When option param :virtual_host is true, the {#bucket} name will be used as the hostname. This will cause the returned URL to be 'http' and not 'https'.

Additional (signed) query parameters can be added to the url by setting option param :query_params to a list of {"key", "value"} pairs. Useful if you are uploading parts of a multipart upload directly from the browser.

Link to this function

put_bucket(bucket, region, opts \\ []) View Source

Creates a bucket in the specified region

Link to this function

put_bucket_acl(bucket, grants) View Source
put_bucket_acl(bucket :: binary(), opts :: [acl_opts()]) ::
  ExAws.Operation.S3.t()

Update or create a bucket bucket access control

Link to this function

put_bucket_cors(bucket, cors_rules) View Source
put_bucket_cors(bucket :: binary(), cors_config :: map()) ::
  ExAws.Operation.S3.t()

Update or create a bucket CORS policy

Link to this function

put_bucket_lifecycle(bucket, livecycle_config) View Source
put_bucket_lifecycle(bucket :: binary(), lifecycle_config :: map()) ::
  no_return()

Update or create a bucket lifecycle configuration

Link to this function

put_bucket_logging(bucket, logging_config) View Source
put_bucket_logging(bucket :: binary(), logging_config :: map()) :: no_return()

Update or create a bucket logging configuration

Link to this function

put_bucket_notification(bucket, notification_config) View Source
put_bucket_notification(bucket :: binary(), notification_config :: map()) ::
  no_return()

Update or create a bucket notification configuration

Link to this function

put_bucket_policy(bucket, policy) View Source
put_bucket_policy(bucket :: binary(), policy :: map()) ::
  ExAws.Operation.S3.t()

Update or create a bucket policy configuration

Link to this function

put_bucket_replication(bucket, replication_config) View Source
put_bucket_replication(bucket :: binary(), replication_config :: map()) ::
  no_return()

Update or create a bucket replication configuration

Link to this function

put_bucket_request_payment(bucket, payer) View Source
put_bucket_request_payment(
  bucket :: binary(),
  payer :: :requester | :bucket_owner
) :: no_return()

Update or create a bucket requestPayment configuration

Link to this function

put_bucket_tagging(bucket, tags) View Source
put_bucket_tagging(bucket :: binary(), tags :: map()) :: no_return()

Update or create a bucket tagging configuration

Link to this function

put_bucket_versioning(bucket, version_config) View Source
put_bucket_versioning(bucket :: binary(), version_config :: binary()) ::
  no_return()

Update or create a bucket versioning configuration

Link to this function

put_bucket_website(bucket, website_config) View Source
put_bucket_website(bucket :: binary(), website_config :: binary()) ::
  no_return()

Update or create a bucket website configuration

Link to this function

put_object(bucket, object, body, opts \\ []) View Source
put_object(
  bucket :: binary(),
  object :: binary(),
  body :: binary(),
  opts :: put_object_opts()
) :: ExAws.Operation.S3.t()

Create an object within a bucket

Link to this function

put_object_acl(bucket, object, acl) View Source
put_object_acl(bucket :: binary(), object :: binary(), acl :: [acl_opts()]) ::
  ExAws.Operation.S3.t()

Create or update an object's access control FIXME

Link to this function

put_object_copy(dest_bucket, dest_object, src_bucket, src_object, opts \\ []) View Source
put_object_copy(
  dest_bucket :: binary(),
  dest_object :: binary(),
  src_bucket :: binary(),
  src_object :: binary(),
  opts :: pub_object_copy_opts()
) :: ExAws.Operation.S3.t()

Copy an object

Link to this function

put_object_tagging(bucket, object, tags, opts \\ []) View Source
put_object_tagging(
  bucket :: binary(),
  object :: binary(),
  tags :: Access.t(),
  opts :: Keyword.t()
) :: ExAws.Operation.S3.t()

Add a set of tags to an existing object

Link to this function

upload(source, bucket, path, opts \\ []) View Source
upload(
  source :: Enumerable.t(),
  bucket :: String.t(),
  path :: String.t(),
  opts :: upload_opts()
) :: ExAws.S3.Upload.t()

Multipart upload to S3.

Handles initialization, uploading parts concurrently, and multipart upload completion.

Uploading a stream

Streams that emit binaries may be uploaded directly to S3. Each binary will be uploaded as a chunk, so it must be at least 5 megabytes in size. The S3.Upload.stream_file helper takes care of reading the file in 5 megabyte chunks.

"path/to/big/file"
|> S3.Upload.stream_file
|> S3.upload("my-bucket", "path/on/s3")
|> ExAws.request! #=> :done

Options

These options are specific to this function

  • :max_concurrency -- The number of concurrent processes reading from this stream. Only applies when uploading a stream.

All other options (ex. :content_type) are passed through to ExAws.S3.initiate_multipart_upload/3.

Link to this function

upload_part(bucket, object, upload_id, part_number, body, opts \\ []) View Source
upload_part(
  bucket :: binary(),
  object :: binary(),
  upload_id :: binary(),
  part_number :: pos_integer(),
  body :: binary(),
  opts :: [encryption_opts() | {:expect, binary()}]
) :: ExAws.Operation.S3.t()

Upload a part for a multipart upload

Link to this function

upload_part_copy(dest_bucket, dest_object, src_bucket, src_object, opts \\ []) View Source
upload_part_copy(
  dest_bucket :: binary(),
  dest_object :: binary(),
  src_bucket :: binary(),
  src_object :: binary(),
  opts :: upload_part_copy_opts()
) :: ExAws.Operation.S3.t()

Upload a part for a multipart copy