ExArrow.Stream (ex_arrow v0.4.0)

View Source

Opaque handle to a native Arrow record-batch stream.

Provides a unified iterator interface over three backing sources:

BackendCreated by
:ipcExArrow.IPC.Reader — Arrow IPC stream or file format
:parquetExArrow.Parquet.Reader — lazy row-group Parquet reader
:adbcExArrow.ADBC.Statement.execute/1 — SQL result streams

Flight do_get results also use the :ipc backend (the Flight client returns an IPC stream resource).

All three backends expose the same three functions:

  • schema/1 — inspect the Arrow schema without consuming any batches
  • next/1 — read the next batch on demand (nil when exhausted)
  • to_list/1 — collect all remaining batches into a list

Record batch data stays in native Arrow memory until consumed. Callers never set the backend field directly; it is assigned by the function that opens the stream.

Summary

Functions

Returns the next record batch from the stream, or nil when done. Returns {:error, message} on read error.

Returns the schema of this stream (without consuming it). Returns {:error, message} if the stream is invalid (e.g. poisoned lock).

Collects all remaining batches from the stream into a list.

Types

t()

@opaque t()

Functions

next(stream)

@spec next(t()) :: ExArrow.RecordBatch.t() | nil | {:error, String.t()}

Returns the next record batch from the stream, or nil when done. Returns {:error, message} on read error.

schema(stream)

@spec schema(t()) :: {:ok, ExArrow.Schema.t()} | {:error, String.t()}

Returns the schema of this stream (without consuming it). Returns {:error, message} if the stream is invalid (e.g. poisoned lock).

to_list(stream)

@spec to_list(t()) :: [ExArrow.RecordBatch.t()]

Collects all remaining batches from the stream into a list.

Stops at the first error and raises. Returns an empty list for an already-exhausted stream.