Module kflow_chunks_to_s3

Description

This module assembles payloads transferred in chunks into S3 objects.

Description

This module assembles payloads transferred in chunks into S3 objects. Warning: chunks should be at least 5MB in size, due to limitations of AWS S3 multipart upload API.

Note: this module aborts multipart uploads in a best effort way. It's still necessary to configure incomplete multipart upload lifecycle to avoid lingering uploads.

Configuration

User needs to specify the bucket where objects will be created:

   #{ s3_bucket := string()
    , prefix    => binary()
    }

Input message format

   #{ key             := binary()
    , value           := binary()
    , headers := [{<<"slice_cnt">>, binary()}, % e.g. <<"42">>
                 ,{<<"slice_num">>, binary()} % e.g. <<"1">> (1-based)
                 ,{<<"deleted">>,<<"0">> | <<"1">>}
                 ]
    , ...
    }

Output message format

   #{ key            := binary()
    , retransmission := boolean()
    , deleted        := boolean()
    , ...
    }

Generated by EDoc