DalaDev.ABTesting (dala_dev v0.2.1)

A/B testing framework for running experiments across mobile device clusters.

Allows you to run experiments, collect metrics, and perform statistical analysis on dala Elixir nodes.

Examples:

# Define an experiment;
experiment = %{
  name: "Cache Strategy Comparison",
  variants: ["strategy_a", "strategy_b"],
  metric: :response_time,
  duration_per_variant: 60_000 # 1 minute per variant
}

# Run the experiment;
{:ok, results} = DalaDev.ABTesting.run(experiment, nodes)

# Analyze results;
{:ok, analysis} = DalaDev.ABTesting.analyze(results)

# Generate report;
DalaDev.ABTesting.generate_report(results, "ab_report.html")

Summary

Types

experiment()

result()

Functions

analyze(results)

Analyze experiment results.

generate_report(results, opts \\ [])

Generate a report from experiment results.

run(experiment, opts \\ [])

Run an A/B test experiment across nodes.

Types

experiment()

@type experiment() :: %{
  name: String.t(),
  variants: [String.t()],
  metric: :response_time | :memory | :reductions | :custom,
  duration_per_variant: integer(),
  warmup: integer(),
  iterations: integer()
}

result()

@type result() :: %{
  variant: String.t(),
  node: node(),
  metric: term(),
  values: [number()],
  stats: map()
}

Functions

analyze(results)

@spec analyze([result()]) :: {:ok, map()} | {:error, term()}

Analyze experiment results.

Returns a map with:

:summary - Overall summary
:variant_stats - Per-variant statistics
:winner - Winning variant (if statistically significant)
:confidence - Confidence level

generate_report(results, opts \\ [])

@spec generate_report(
  [result()],
  keyword()
) :: {:ok, String.t() | :ok} | {:error, term()}

Generate a report from experiment results.

Options:

:format - :html (default) or :text
:output - Output file path (optional)

Returns the report content or :ok if saved.

run(experiment, opts \\ [])

@spec run(
  experiment(),
  keyword()
) :: {:ok, [result()]} | {:error, term()}

Run an A/B test experiment across nodes.

Options:

:nodes - List of nodes to run experiment on
:iterations - Number of iterations per variant (default: 10)
:warmup - Warmup iterations (default: 3)
:timeout - RPC timeout in ms (default: 60_000)

Returns a list of result maps.