macula_content_chunker (macula v0.20.5)
View SourceContent chunking module for Macula content-addressed storage.
Provides functions to split binary data into fixed-size chunks, reassemble chunks back into the original data, and compute Merkle tree root hashes for verification.
Chunk Size
The default chunk size is 256KB (262144 bytes), which provides a good balance between: - Network efficiency (not too many small packets) - Memory usage (chunks fit in L2 cache) - Parallelism (enough chunks for multi-provider download)
Example Usage
%% Split data into chunks
{ok, Chunks} = macula_content_chunker:chunk(Data, 262144),
%% Get chunk metadata with hashes
Infos = macula_content_chunker:chunk_info(Chunks, sha256),
%% Compute Merkle root
Root = macula_content_chunker:merkle_root(Infos, sha256),
%% Reassemble chunks
Data = macula_content_chunker:reassemble(Chunks).Summary
Functions
Split binary data into fixed-size chunks. Returns a list of binaries, where all but the last are exactly ChunkSize bytes. The last chunk may be smaller. Empty input returns an empty list.
Create chunk info records with index, offset, size, and hash.
Return the default chunk size (256KB).
Compute Merkle tree root hash from chunk infos. Uses a binary tree structure where leaf nodes are chunk hashes and internal nodes are hash(left || right).
Reassemble chunks back into original data.
Verify a chunk against its expected hash.
Types
-type chunk_info() :: #{index := non_neg_integer(), offset := non_neg_integer(), size := pos_integer(), hash := binary()}.
Functions
-spec chunk(binary(), pos_integer()) -> {ok, [binary()]}.
Split binary data into fixed-size chunks. Returns a list of binaries, where all but the last are exactly ChunkSize bytes. The last chunk may be smaller. Empty input returns an empty list.
-spec chunk_info([binary()], macula_content_hasher:algorithm()) -> [chunk_info()].
Create chunk info records with index, offset, size, and hash.
-spec default_chunk_size() -> pos_integer().
Return the default chunk size (256KB).
-spec merkle_root([chunk_info()], macula_content_hasher:algorithm()) -> binary().
Compute Merkle tree root hash from chunk infos. Uses a binary tree structure where leaf nodes are chunk hashes and internal nodes are hash(left || right).
Reassemble chunks back into original data.
-spec verify_chunk(binary(), binary(), macula_content_hasher:algorithm()) -> boolean().
Verify a chunk against its expected hash.