HLL

v0.1.1

HLL v0.1.1 HLL.Redis View Source

Redis compatible HyperLogLog module.

This module is Redis (v5) compatible. It uses the same hash algorithm, same HyperLogLog estimation algorithm and same serialization format as Redis (v5) does.

Therefore, it could consume HyperLogLog sketches from Redis and it could generate HyperLogLog sketches for Redis as well.

It has fixed precision 14 (16384 buckets) as Redis does. If you are looking for using different precision, you could use HLL module instead.

HLL.Redis module is generally slower than alternative HLL module:

  • HLL.Redis hash function is slower: Hash function in HLL.Redis is ported from Redis and written in Elixir. Hash function in HLL is :erlang.phash2, which is in native code.
  • HLL.Redis serialization is slower: HLL.Redis uses Redis binary format for serialization. HLL uses a binary format which is closer to HLL's internal data structure, which makes it faster to encode and decode.

Therefore, if you do not require "Redis compatible", it's recommanded to use HLL module for performance gain.

Example 

iex> hll_redis = HLL.Redis.new()
iex> hll_redis = Enum.reduce(1..2000, hll_redis, fn i, acc -> HLL.Redis.add(acc, Integer.to_string(i)) end)
iex> HLL.Redis.cardinality(hll_redis)
2006

Link to this section Summary

Types

t()

Functions

add(hll_redis, item)

Add a value to Redis compatible HyperLogLog instance

cardinality(hll_redis)

Estimate cardinality of Redis compatible instance

decode(redis_binary)

Decode Redis HyperLogLog binary format to Redis compatible HyperLogLog instance

encode(hll_redis)

Encode Redis compatible HyperLogLog instance to Redis HyperLogLog binary format

merge(list_of_hll_redis)

Merge multiple Redis compatible HyperLogLog instances into one

new()

Create a Redis compatible HyperLogLog instance with precision = 14

Link to this section Types

Link to this type

t() View Source 
t() :: {HLL.Redis, map()}

Link to this section Functions

Link to this function

add(hll_redis, item) View Source 
add(t(), any()) :: t()

Add a value to Redis compatible HyperLogLog instance.

If item is binary, it would use Redis compatible murmur2 hash function directly.

If item is not binary, it would be transformed to binary via :erlang.term_to_binary/1 and then use Redis compatible murmur2 hash function.

Example 

iex> HLL.Redis.new() |> HLL.Redis.add("hello")
{HLL.Redis, %{9216 => 1}}
Link to this function

cardinality(hll_redis) View Source 
cardinality(t()) :: non_neg_integer()

Estimate cardinality of Redis compatible instance.

Example 

iex> data = Enum.map(1..5000, &Integer.to_string/1)
iex> h = HLL.Redis.new()
iex> h = Enum.reduce(data, h, fn x, acc -> HLL.Redis.add(acc, x) end)
iex> HLL.Redis.cardinality(h)
4985
iex> {:ok, conn} = Redix.start_link()
iex> for x <- data do Redix.command!(conn, ["PFADD", "test_hll_redis_cardinality", x]) end
iex> Redix.command!(conn, ["PFCOUNT", "test_hll_redis_cardinality"])
4985
Link to this function

decode(redis_binary) View Source 
decode(binary()) :: t()

Decode Redis HyperLogLog binary format to Redis compatible HyperLogLog instance.

Example 

iex> {:ok, conn} = Redix.start_link()
iex> Redix.command!(conn, ["PFADD", "test_hll_redis_decode", "okk"])
iex> bin = Redix.command!(conn, ["GET", "test_hll_redis_decode"])
<<72, 89, 76, 76, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 108, 180, 132, 83, 73>>
iex> HLL.Redis.decode(bin)
{HLL.Redis, %{11445 => 2}}
iex> HLL.Redis.new() |> HLL.Redis.add("okk")
{HLL.Redis, %{11445 => 2}}
Link to this function

encode(hll_redis) View Source 
encode(t()) :: binary()

Encode Redis compatible HyperLogLog instance to Redis HyperLogLog binary format.

Example 

iex> HLL.Redis.new() |> HLL.Redis.add("hello") |> HLL.Redis.encode()
<<72, 89, 76, 76, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 99, 255, 128, 91, 254>>
iex> {:ok, conn} = Redix.start_link()
iex> Redix.command!(conn, ["PFADD", "test_hll_redis_encode", "hello"])
iex> Redix.command!(conn, ["GET", "test_hll_redis_encode"])
<<72, 89, 76, 76, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 99, 255, 128, 91, 254>>
Link to this function

merge(list_of_hll_redis) View Source

Merge multiple Redis compatible HyperLogLog instances into one.

Example 

iex> h1 = HLL.Redis.new() |> HLL.Redis.add("foo")
iex> h2 = HLL.Redis.new() |> HLL.Redis.add("bar")
iex> h3 = HLL.Redis.new() |> HLL.Redis.add("foo") |> HLL.Redis.add("bar")
iex> h_merged = HLL.Redis.merge([h1, h2])
iex> h3 == h_merged
true
Link to this function

new() View Source 
new() :: t()

Create a Redis compatible HyperLogLog instance with precision = 14.

Example 

iex> HLL.Redis.new()
{HLL.Redis, %{}}