SqlKit.DuckDB (sql_kit v0.2.0)

View Source

DuckDB support for SqlKit.

Provides two ways to use DuckDB with SqlKit:

Direct Connection (BYO)

For simple use cases, scripts, or explicit control:

{:ok, conn} = SqlKit.DuckDB.connect(":memory:")
SqlKit.query_all(conn, "SELECT 1 as num", [])
# => [%{num: 1}]
SqlKit.DuckDB.disconnect(conn)

For production use, add the pool to your supervision tree:

children = [
  {SqlKit.DuckDB.Pool,
    name: MyApp.AnalyticsPool,
    database: "priv/analytics.duckdb",
    pool_size: 4}
]

# Then use the pool name with SqlKit functions
SqlKit.query_all(MyApp.AnalyticsPool, "SELECT * FROM events", [])

Loading Extensions

DuckDB extensions are loaded via SQL (SQL-first philosophy):

SqlKit.query!(conn, "INSTALL 'parquet';", [])
SqlKit.query!(conn, "LOAD 'parquet';", [])
SqlKit.query_all(conn, "SELECT * FROM 'data.parquet'", [])

Notes

  • Uses PostgreSQL-style $1, $2, ... parameter placeholders
  • In-memory database: use ":memory:" string (not :memory atom)
  • Hugeint values are automatically converted to Elixir integers

Summary

Functions

Opens a DuckDB database and creates a connection.

Opens a DuckDB database and creates a connection. Raises on error.

Converts hugeint tuples to integers in result rows.

Converts a single value, handling hugeint tuples.

Closes a DuckDB connection and releases the database.

Executes a SQL query and returns columns and rows.

Executes a SQL query and returns columns and rows. Raises on error.

Executes a SQL query and returns a stream of result chunks.

Like stream!/3 but also returns column names.

Types

connect_opts()

@type connect_opts() :: [{:config, struct()}]

Functions

connect(database, opts \\ [])

@spec connect(String.t(), connect_opts()) ::
  {:ok, SqlKit.DuckDB.Connection.t()} | {:error, term()}

Opens a DuckDB database and creates a connection.

Arguments

  • database - Path to database file or ":memory:" for in-memory database

Options

Examples

# In-memory database
{:ok, conn} = SqlKit.DuckDB.connect(":memory:")

# File-based database
{:ok, conn} = SqlKit.DuckDB.connect("analytics.duckdb")

# With configuration
{:ok, conn} = SqlKit.DuckDB.connect("analytics.duckdb",
  config: %Duckdbex.Config{threads: 4})

connect!(database, opts \\ [])

Opens a DuckDB database and creates a connection. Raises on error.

See connect/2 for options.

convert_hugeints(rows)

@spec convert_hugeints([[term()]]) :: [[term()]]

Converts hugeint tuples to integers in result rows.

Duckdbex represents HUGEINT (128-bit integers) as {upper, lower} tuples. This function recursively converts all such tuples to Elixir integers.

This is safe because other duckdbex tuple types have different arities:

  • DATE: {year, month, day}
  • TIME: {hour, minute, second, microsecond}
  • DECIMAL: {value, precision, scale}
  • TIMESTAMP: {{y, m, d}, {h, m, s, us}}

If duckdbex adds another 2-integer-tuple type in the future, this would need to be updated. Check duckdbex changelog on upgrades.

convert_value(value)

@spec convert_value(term()) :: term()

Converts a single value, handling hugeint tuples.

See convert_hugeints/1 for details.

disconnect(connection)

@spec disconnect(SqlKit.DuckDB.Connection.t()) :: :ok

Closes a DuckDB connection and releases the database.

Examples

{:ok, conn} = SqlKit.DuckDB.connect(":memory:")
:ok = SqlKit.DuckDB.disconnect(conn)

query(connection, sql, params)

@spec query(SqlKit.DuckDB.Connection.t(), String.t(), list()) ::
  {:ok, {[String.t()], [[term()]]}} | {:error, term()}

Executes a SQL query and returns columns and rows.

This is a low-level function. Users should typically use SqlKit.query_all/3, SqlKit.query_one!/3, etc. instead.

Examples

{:ok, {columns, rows}} = SqlKit.DuckDB.query(conn, "SELECT 1 as num", [])
# => {:ok, {["num"], [[1]]}}

query!(conn, sql, params)

@spec query!(SqlKit.DuckDB.Connection.t(), String.t(), list()) ::
  {[String.t()], [[term()]]}

Executes a SQL query and returns columns and rows. Raises on error.

See query/3 for details.

stream!(connection, sql, params)

Executes a SQL query and returns a stream of result chunks.

Unlike query!/3 which loads all results into memory, stream!/3 returns a lazy Stream that fetches results in chunks. This is useful for large result sets that would otherwise consume too much memory.

Each chunk is a list of rows (each row is a list of values). Use Stream.flat_map/2 to iterate over individual rows.

Examples

# Stream large result set
conn
|> SqlKit.DuckDB.stream!("SELECT * FROM large_table", [])
|> Stream.flat_map(& &1)
|> Enum.take(100)

# With column names (first element of tuple)
{columns, row_stream} = SqlKit.DuckDB.stream_with_columns!(conn, sql, [])
rows = row_stream |> Stream.flat_map(& &1) |> Enum.to_list()

Notes

  • The connection is held for the duration of stream consumption
  • Hugeint values are automatically converted to Elixir integers
  • For pooled connections, use SqlKit.DuckDB.Pool.with_stream!/5

stream_with_columns!(connection, sql, params)

@spec stream_with_columns!(SqlKit.DuckDB.Connection.t(), String.t(), list()) ::
  {[String.t()], Enumerable.t()}

Like stream!/3 but also returns column names.

Returns {columns, row_chunk_stream} where columns is a list of column names and row_chunk_stream is a stream of row chunks.

Examples

{columns, stream} = SqlKit.DuckDB.stream_with_columns!(conn, "SELECT * FROM users", [])
# columns => ["id", "name", "age"]
rows = stream |> Stream.flat_map(& &1) |> Enum.to_list()