Quant.Explorer.DataTransformer (quant v0.1.0-alpha.1)

Data normalization utilities for converting raw API responses into standardized Explorer DataFrames.

This module handles the transformation of various data formats (JSON, CSV, maps, lists) from different providers into consistent DataFrame schemas that can be used across the entire library.

Summary

Functions

Filters out rows with invalid or missing critical data.

Converts CSV string to DataFrame with proper column types.

Normalizes numeric values, handling strings, floats, integers, and nil.

Normalizes timestamps from various formats to DateTime.

Normalizes volume values, typically as integers.

Transforms raw historical data into a standardized DataFrame.

Transforms raw quote data into a standardized DataFrame.

Transforms raw search results into a standardized DataFrame.

Types

raw_data()

@type raw_data() :: map() | list() | binary()

schema()

@type schema() :: :history | :quote | :search | :info

Functions

clean_dataframe(df, arg2)

@spec clean_dataframe(Explorer.DataFrame.t(), schema()) :: Explorer.DataFrame.t()

Filters out rows with invalid or missing critical data.

csv_to_dataframe(csv_data, opts \\ [])

@spec csv_to_dataframe(
  binary(),
  keyword()
) :: {:ok, Explorer.DataFrame.t()} | {:error, term()}

Converts CSV string to DataFrame with proper column types.

normalize_number(n)

@spec normalize_number(term()) :: float() | nil

Normalizes numeric values, handling strings, floats, integers, and nil.

normalize_timestamp(dt)

@spec normalize_timestamp(term()) :: DateTime.t() | nil

Normalizes timestamps from various formats to DateTime.

Handles:

  • Unix timestamps (seconds/milliseconds)
  • ISO 8601 strings
  • Date strings in various formats
  • Already parsed DateTime structs

normalize_volume(n)

@spec normalize_volume(term()) :: integer() | nil

Normalizes volume values, typically as integers.

transform_history(data, symbol, opts \\ [])

@spec transform_history(raw_data(), String.t(), keyword()) ::
  {:ok, Explorer.DataFrame.t()} | {:error, term()}

Transforms raw historical data into a standardized DataFrame.

Expected columns in output:

  • symbol (string): Stock/crypto symbol
  • timestamp (datetime): Data timestamp
  • open (f64): Opening price
  • high (f64): High price
  • low (f64): Low price
  • close (f64): Closing price
  • volume (s64): Trading volume
  • adj_close (f64): Adjusted closing price (optional)

transform_quote(data, symbols, opts \\ [])

@spec transform_quote(raw_data(), String.t() | [String.t()], keyword()) ::
  {:ok, Explorer.DataFrame.t()} | {:error, term()}

Transforms raw quote data into a standardized DataFrame.

Expected columns in output:

  • symbol (string): Stock/crypto symbol
  • price (f64): Current price
  • change (f64): Price change
  • change_percent (f64): Percentage change
  • volume (s64): Current volume
  • timestamp (datetime): Quote timestamp

transform_search(data, opts \\ [])

@spec transform_search(
  raw_data(),
  keyword()
) :: {:ok, Explorer.DataFrame.t()} | {:error, term()}

Transforms raw search results into a standardized DataFrame.

Expected columns in output:

  • symbol (string): Trading symbol
  • name (string): Company/asset name
  • type (string): Asset type (stock, etf, crypto, etc.)
  • exchange (string): Trading exchange (optional)