Pipeline.Data.Transformer (pipeline v0.0.1)

View Source

Data transformation engine for manipulating structured data between pipeline steps.

Supports operations:

  • filter: Filter items based on conditions
  • map: Transform each item
  • aggregate: Aggregate values (sum, average, count, etc.)
  • join: Join with data from another source
  • group_by: Group items by field values
  • sort: Sort items by field values
  • transform: Apply custom transformations

Uses JSONPath-like syntax for field access and supports chaining operations.

Summary

Types

operation()

Functions

apply_operation(data, operation, context \\ %{})

Apply a single transformation operation.

transform(data, operations, context \\ %{})

Apply a series of transformation operations to data.

Types

operation()

@type operation() :: %{
  :operation => String.t(),
  optional(:field) => String.t(),
  optional(:condition) => String.t(),
  optional(:function) => String.t(),
  optional(:value) => any(),
  optional(:mapping) => map(),
  optional(:left_field) => String.t(),
  optional(:right_source) => String.t(),
  optional(:join_key) => String.t(),
  optional(:order) => String.t(),
  optional(:expression) => String.t()
}

Functions

apply_operation(data, operation, context \\ %{})

@spec apply_operation(any(), operation(), map()) ::
  {:ok, any()} | {:error, String.t()}

Apply a single transformation operation.

transform(data, operations, context \\ %{})

@spec transform(any(), [operation()], map()) :: {:ok, any()} | {:error, String.t()}

Apply a series of transformation operations to data.