Module kflow_chunks_to_file

Description
Function Index
Function Details

This module assembles payloads transferred in chunks into files.

Description

This module assembles payloads transferred in chunks into files. Empty files aren't created.

Configuration

User needs to specify the location where files will be created.

   #{location := file:filename_all()}

Input message format

   #{ key             := binary()
    , value           := binary()
    , headers := [{<<"slice_cnt">>, binary()}, % e.g. <<"42">>
                 ,{<<"slice_num">>, binary()} % e.g. <<"1">> (1-based)
                 ,{<<"deleted">>,<<"0">> | <<"1">>}
                 ]
    , ...
    }

Output message format

   #{ key            := binary()
    , retransmission := boolean()
    , deleted        := boolean()
    }

Error handling

This module is designed to survive the following permutations of the messages:

Restart of the upstream from the beginning: [1, 2, 3, 1, 2, 3, 4, 5]
Restart of the upstream in the middle of transfer: [1, 2, 3, 2, 3, 4, 5]
Retransmission from the middle: [3, 4, 5]

This allows this pipe to survive restarts of the upstream. Duplicate chunks are ignored.

Function Index

chunk_count/1
chunk_num/1
in/6
out/2

Function Details

chunk_count/1

chunk_count(X1) -> any()

chunk_num/1

chunk_num(X1) -> any()

in/6

in(Offset, SliceNum, SliceCnt, Msg, State, Config) -> any()

out/2

out(State, X2) -> any()

Generated by EDoc