SparkEx.StreamingQuery (SparkEx v0.1.0)

Controls a running structured streaming query.

Mirrors PySpark's StreamingQuery with methods for monitoring and controlling the query lifecycle.

Examples

{:ok, query} = writer |> SparkEx.StreamWriter.start()
{:ok, true} = SparkEx.StreamingQuery.is_active?(query)
:ok = SparkEx.StreamingQuery.stop(query)

Summary

Types

t()

Functions

await_termination(query, opts \\ [])

Waits for the termination of the query.

exception(query)

Returns exception information if the query has terminated with an error.

explain(query, opts \\ [])

Returns the query plan as a string.

id(query)

Returns the streaming query ID.

is_active?(query)

Returns whether the streaming query is currently active.

last_progress(query)

Returns the most recent progress report as a JSON string, or nil.

name(query)

Returns the streaming query name.

process_all_available(query)

Blocks until all available data in the source has been processed.

recent_progress(query)

Returns a list of recent progress reports as JSON strings.

run_id(query)

Returns the streaming query run ID.

status(query)

Returns the current status of the streaming query.

stop(query)

Stops the streaming query.

Types

t()

@type t() :: %SparkEx.StreamingQuery{
  name: String.t() | nil,
  query_id: String.t(),
  run_id: String.t(),
  session: GenServer.server()
}

Functions

await_termination(query, opts \\ [])

@spec await_termination(
  t(),
  keyword()
) :: {:ok, boolean() | nil} | {:error, term()}

Waits for the termination of the query.

Options

:timeout — timeout in milliseconds (default: no timeout)

exception(query)

@spec exception(t()) :: {:ok, map() | nil} | {:error, term()}

Returns exception information if the query has terminated with an error.

explain(query, opts \\ [])

@spec explain(
  t(),
  keyword()
) :: {:ok, String.t()} | {:error, term()}

Returns the query plan as a string.

Options

:extended — whether to include extended details (default: false)

id(query)

@spec id(t()) :: String.t()

Returns the streaming query ID.

is_active?(query)

@spec is_active?(t()) :: {:ok, boolean()} | {:error, term()}

Returns whether the streaming query is currently active.

last_progress(query)

@spec last_progress(t()) :: {:ok, String.t() | nil} | {:error, term()}

Returns the most recent progress report as a JSON string, or nil.

name(query)

@spec name(t()) :: String.t() | nil

Returns the streaming query name.

process_all_available(query)

@spec process_all_available(t()) :: :ok | {:error, term()}

Blocks until all available data in the source has been processed.

recent_progress(query)

@spec recent_progress(t()) :: {:ok, [String.t()]} | {:error, term()}

Returns a list of recent progress reports as JSON strings.

run_id(query)

@spec run_id(t()) :: String.t()

Returns the streaming query run ID.

status(query)

@spec status(t()) :: {:ok, map()} | {:error, term()}

Returns the current status of the streaming query.

stop(query)

@spec stop(t()) :: :ok | {:error, term()}

Stops the streaming query.