PropertyDamage.Flakiness (PropertyDamage v0.1.0)

Detect non-deterministic behavior in the system under test.

Flaky tests are tests that sometimes pass and sometimes fail with the same input. This is often caused by:

Race conditions in the SUT
External dependencies (time, network, etc.)
Uninitialized state between runs
Non-deterministic SUT behavior

Usage

# Check if a specific seed is flaky
result = PropertyDamage.check_determinism(
  model: M,
  adapter: A,
  seed: 512902757,
  runs: 10
)

case result do
  {:ok, :deterministic} ->
    IO.puts("Seed is deterministic")

  {:ok, :flaky, stats} ->
    IO.puts("Seed is FLAKY: passed #{stats.passes}/#{stats.runs} times")

  {:error, reason} ->
    IO.puts("Check failed: #{inspect(reason)}")
end

Batch Checking

# Check multiple seeds at once
results = PropertyDamage.check_determinism_batch(
  model: M,
  adapter: A,
  seeds: [123, 456, 789],
  runs_per_seed: 5
)

Understanding Results

The checker reports:

Deterministic: Same result every run
Flaky: Different results across runs
Variance: What varies (pass/fail, failure type, shrunk size)

Summary

Types

check_opts()

flaky_stats()

result()

Functions

check(model, adapter, seed, opts \\ [])

Check if a seed produces deterministic results.

check_batch(model, adapter, seeds, opts \\ [])

Check multiple seeds for flakiness.

discover_flaky(model, adapter, opts \\ [])

Run random seeds and identify flaky ones.

format_batch(results)

Format batch results for display.

format_result(arg)

Format flakiness check results for display.

Types

check_opts()

@type check_opts() :: [
  runs: non_neg_integer(),
  adapter_config: map(),
  max_commands: non_neg_integer(),
  verbose: boolean()
]

flaky_stats()

@type flaky_stats() :: %{
  runs: non_neg_integer(),
  passes: non_neg_integer(),
  failures: non_neg_integer(),
  failure_types: %{required(atom()) => non_neg_integer()},
  shrunk_sizes: [non_neg_integer()],
  variance_type: :pass_fail | :failure_type | :shrunk_size | :multiple
}

result()

@type result() ::
  {:ok, :deterministic} | {:ok, :flaky, flaky_stats()} | {:error, term()}

Functions

check(model, adapter, seed, opts \\ [])

@spec check(module(), module(), integer(), check_opts()) :: result()

Check if a seed produces deterministic results.

Runs the same seed multiple times and compares outcomes.

Options

:runs - Number of times to run (default: 5)
:adapter_config - Adapter configuration
:max_commands - Maximum commands per run (default: 50)
:verbose - Print progress (default: false)

Returns

{:ok, :deterministic} - Same result every time
{:ok, :flaky, stats} - Different results, with statistics
{:error, reason} - Check failed

check_batch(model, adapter, seeds, opts \\ [])

@spec check_batch(module(), module(), [integer()], keyword()) :: %{
  required(integer()) => result()
}

Check multiple seeds for flakiness.

Options

Same as check/4, plus:

:runs_per_seed - Runs per seed (default: 5)
:parallel - Run seeds in parallel (default: false)

Returns

Map from seed to result.

discover_flaky(model, adapter, opts \\ [])

@spec discover_flaky(module(), module(), keyword()) :: [{integer(), flaky_stats()}]

Run random seeds and identify flaky ones.

This is useful for discovering non-determinism in your SUT without knowing specific problematic seeds.

Options

:num_seeds - Number of random seeds to test (default: 10)
:runs_per_seed - Runs per seed (default: 3)
Other options passed to check/4

Returns

List of {seed, flaky_stats} for seeds that are flaky.

format_batch(results)

@spec format_batch(%{required(integer()) => result()}) :: String.t()

Format batch results for display.

format_result(arg)

@spec format_result(result()) :: String.t()

Format flakiness check results for display.