View Source ExZstd (ex_zstd v0.5.2)
ExZstd aims to expose ZSTD stable interfaces in an idiomatic Elixir way. If you have any doubts the official documentation resides here: https://github.com/facebook/zstd/blob/v1.5.5/lib/zstd.h. The naming used in this wrapper library aims to reflect upstream closely where possible.
Simple API
99% of users only needs this API, it's composed by two methods: compress
and decompress
that do what they say on the tin.
Note that decompress reads the final size from the header, so only pass trusted input to this function. To decompress untrused input see
decompress_stream
under the stream API section.
Context API
This API allows the caller to reuse the same context for multiple compression and specify advanced parameters, see the zstd.h
header to
know more. Your code will probably looks simlar to the follwing:
alias ExZstd.CCtx
alias ExZstd.Cparam
alias ExZstd.Cparam.Strategy
alias ExZstd.DCtx
data = :crypto.strong_rand_bytes(16)
cctx = CCtx.create()
|> CCtx.set_cparam(Cparam.compression_level, 3)
|> CCtx.set_cparam(Cparam.strategy, Strategy.btopt)
compressed_data = ExZstd.CCtx.compress!(cctx, data)
dctx = DCtx.create()
original_data = ExZstd.DCtx.decompress(dctx, compressed_data)
data == original_data
Important note: the various context API that set/get parameters will modify the underlying context object, not create a new one. So the two following snippets both modify cctx
:
# first
cctx = CCtx.new()
CCtx.set_cparam(cctx, param.compression_level, 3)
#second
cctx = CCtx.new() |> CCtx.set_cparam(Cparam.compression_level, 3)
Threads
- 1 context = 1 thread AND no simultaneous compressions
- If you pass the right parameters you'll be able to use the multithreading capabilities of zstd. The thread pool would live outside the BEAM so avoid it unless absolutely necessary.
Stream API
The stream API allows you to compress and decompress multiple data in frames lazily. Use this api if:
- you want to compress complex data (e.g. iolists) without transforming it into a binary first.
- you want to decompress untrusted data
- you want to control how the input is split into multiple frames
Example:
alias ExZstd.Cctx
alias ExZstd.Dctx
alias ExZstd.Cparam
alias ExZstd.Dparam
cctx = CCtx.create()
dctx = DCtx.create()
# compress the data
first_chunk = CCtx.compress_stream!(cctx, Fixtures.data(), CCtx.stream_continue())
second_chunk = CCtx.compress_stream!(cctx, Fixtures.data2(), CCtx.stream_end())
# retreive the input
DCtx.decompress_stream!(dctx, first_chunk) <> DCtx.decompress_stream!(dctx, second_chunk)
Dictionary API
You can use the dictionary API to further improve compression size.
Excerpts from the zdict.h
header:
- In general, it's recommended to provide a few thousands samples, though this can vary a lot.
- It's recommended that total size of all samples be about ~x100 times the target size of dictionary.
You can train the dictionaries by using the zstd --train
interface of the CLI. At the moment this interface is not tested as part of ex_zstd
test suite.
Example:
alias ExZstd.CCtx
alias ExZstd.Cparam
alias ExZstd.Cparam.Strategy
alias ExZstd.DCtx
data = :crypto.strong_rand_bytes(16)
dictionary = File.read!("my_dictionary.bin")
cctx = CCtx.create()
CCtx.load_dictionary(dictionary)
compressed_data = ExZstd.CCtx.compress!(cctx, data)
dctx = DCtx.create()
DCtx.load_dictionary(dictionary)
original_data = ExZstd.DCtx.decompress(dctx, compressed_data)
data == original_data
Summary
Functions
Identical to compress!
. Wraps the result in a maybe instead of raising.
Compresses a chunk of data.
Returns ZSTD_CONTENTSIZE_UNKNOWN.
Identical to decompress!
. Wraps the result in a maybe instead of raising.
Decompresses a chunk of data. If :content_size_unknown
is returned stream decompression can be still attempted.
Returns the boundaries for a compresson parameter.
Returns the boundaries of a decompression parameter.
Types
Functions
Identical to compress!
. Wraps the result in a maybe instead of raising.
Compresses a chunk of data.
@spec content_size_unknown() :: pos_integer()
Returns ZSTD_CONTENTSIZE_UNKNOWN.
@spec decompress(binary()) :: maybe(binary(), decompression_error())
Identical to decompress!
. Wraps the result in a maybe instead of raising.
Note: this functions reads the header to find out the size of the decompressed data. Only feed it trusted input!
Decompresses a chunk of data. If :content_size_unknown
is returned stream decompression can be still attempted.
Note: this functions reads the header to find out the size of the decompressed data. Only feed it trusted input!
@spec default_compression_level() :: 3
@spec get_cparam_bounds(pos_integer()) :: {integer(), integer()}
Returns the boundaries for a compresson parameter.
@spec get_dparam_bounds(pos_integer()) :: {integer(), integer()}
Returns the boundaries of a decompression parameter.
@spec reset_parameters() :: 2
@spec reset_session_and_parameters() :: 3
@spec reset_session_only() :: 1