Changelog

Copy Markdown View Source

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.6.0 - 2026-03-02

Added

  • TelemetryHelpers.detach_isolated/1 for explicit teardown of isolated telemetry handlers and associated buffers.
  • TelemetryHelpers.flush_buffered_telemetry/1 for draining buffered telemetry events by handler ID.
  • Internal module Supertester.Internal.Poller — unified deadline-based polling utility replacing hand-rolled wait_for_*_loop patterns across the codebase.
  • Internal module Supertester.Internal.ProcessLifecycle — consolidated process stop/cleanup logic with :shutdown escalation to :kill and unlinking before stop.
  • Internal module Supertester.Internal.ProcessRef — unified name resolution for pid, atom, {:global, _}, and {:via, _, _} references.
  • Internal module Supertester.Internal.SharedRegistry — single shared Registry (:supertester_shared_registry) for all test process name registration, replacing per-test registries.
  • Internal module Supertester.Internal.SupervisorIntrospection — shared supervisor introspection utilities (extract_supervisor_strategy, extract_supervisor_module, resolve_supervisor_pid, group_child_pids_by_id) extracted from Assertions, SupervisorHelpers, and ChaosHelpers.
  • Internal module Supertester.Internal.IsolationContextStore — consolidated maybe_update_context/1 pattern shared by ETSIsolation, LoggerIsolation, and TelemetryHelpers.
  • Internal module Supertester.Internal.TelemetryBuffer — anonymous Agent-based telemetry event buffer replacing per-process dynamically-named agents.
  • Internal module Supertester.Internal.TelemetryHandlerBuffers — process dictionary manager mapping handler IDs to buffer pids.
  • Internal module Supertester.Internal.TelemetryMailbox — extracted telemetry mailbox receive/flush/collect logic from TelemetryHelpers.
  • Internal module Supertester.Internal.Chaos.ResourceExhaustion — extracted resource exhaustion simulation logic from ChaosHelpers.
  • Internal module Supertester.Internal.Chaos.SuiteRunner — timeout-aware chaos suite execution with per-scenario deadline enforcement.
  • Internal module Supertester.ConcurrentHarness.Runtime — execution engine extracted from ConcurrentHarness (~420 lines).
  • Comprehensive test suites for Assertions, ChaosHelpers edge cases, ConcurrentHarness API contract stability, ETSIsolation cleanup behavior, and all new internal modules (Poller, ProcessLifecycle, ProcessRef, SharedRegistry, SupervisorIntrospection, TelemetryBuffer).
  • Atom safety regression tests verifying zero unbounded atom creation during isolation setup, ETS exhaustion simulation, and concurrent telemetry usage.
  • Regression tests for cast_and_sync/4 function-clause missing-sync handling, temporary-child restart reporting, :one_for_all cascade restart accounting, ordinary :timeout scenario failure handling in run_chaos_suite/3, and short-lived transient process handling in assert_no_process_leaks/1.
  • Supertester.Internal.IsolationContextStore.put_updated/2 to consolidate context mutation + persistence in one internal helper.
  • Supertester.Internal.SupervisorIntrospection.safe_children/1, replacement_count/2, and child_restarted?/2 for shared supervisor-state/restart analysis across helpers.
  • Targeted regression tests for shared registry stale-name/reset recovery, memory trend analysis behavior, message trace timeout handling, and isolation context store update semantics.
  • Internal module Supertester.Internal.ProcessWatch — shared monitor/DOWN wait utility used across lifecycle-sensitive internals.
  • Focused internal test coverage for TelemetryMailbox selective flush/requeue behavior and ProcessWatch timeout/down semantics.
  • Cleanup regression coverage for mailbox monitor tasks on raise/throw/exit, concurrent-harness timeout cleanup, and shared-registry lifecycle stability under repeated isolation setup.

Changed

  • GenServerHelpers.cast_and_sync/4: Non-strict mode now consistently returns {:error, :missing_sync_handler} instead of :ok when sync handling is missing. Any successful sync call reply (including {:error, :unknown_call}) is treated as synchronized and returns {:ok, reply}.
  • SupervisorHelpers.test_restart_strategy/3: Raises ArgumentError when scenario child IDs are missing or when the supervisor's actual strategy does not match the expected strategy.
  • SupervisorHelpers.assert_supervision_tree_structure/2: Now validates expected child modules for leaf children and optional :supervisor/:strategy keys in the expected structure.
  • SupervisorHelpers.wait_for_supervisor_stabilization/2: Now delegates directly to UnifiedTestFoundation.wait_for_supervision_tree_ready/2 (flattened wrapper chain).
  • Assertions.assert_supervisor_strategy/2: Now validates runtime supervisor strategy via :sys.get_state/1 introspection instead of only verifying process accessibility. Handles both tuple-based (standard Supervisor) and map-based (DynamicSupervisor) internal state formats.
  • Assertions.assert_no_process_leaks/1: Uses :erlang.trace/3 to scope leak detection to processes spawned/linked by the operation caller, with a post-operation grace window for delayed descendant detection. Replaces global Process.list() snapshot comparison.
  • ChaosHelpers.chaos_kill_children/2: Treats kill_rate <= 0 as a no-kill run. Restart accounting now includes observed cascade replacements (e.g., :one_for_all sibling restarts).
  • ChaosHelpers.run_chaos_suite/3: Accepts timeout: option for suite-wide deadline enforcement. Distinguishes true deadline overruns from ordinary scenario failures with reason :timeout.
  • ChaosHelpers.simulate_resource_exhaustion/2: Non-positive spawn_count/count values treated as no-op instead of allocating resources via descending ranges. ETS table names use a single reusable atom instead of per-table dynamic atoms.
  • ETSIsolation.inject_table/3-4: Fallback path rejects dynamic atom creation; requires a pre-existing env key or __supertester_set_table__/2 callback. Restore is now idempotent via injection tracking.
  • OTPHelpers.safe_start/4: Uses a spawned intermediary process for start_link calls instead of setting trap_exit on the test process, preventing side effects on ExUnit's process monitoring.
  • OTPHelpers process naming: Switched from dynamic atom creation (String.to_atom/1) to {:via, Registry, {:supertester_shared_registry, ...}} tuples for zero-atom-allocation naming. Single shared Registry replaces per-test registries.
  • UnifiedTestFoundation.generate_test_id/1: Returns a string instead of an atom.

  • IsolationContext.t(): :name and :registry fields broadened from atom() | nil to term() | nil to support {:via, ...} tuples.

  • ExUnitFoundation.__using__/1: Setup logic extracted to named functions (maybe_setup_telemetry/2, maybe_setup_logger/3, maybe_setup_ets/2) instead of being inlined in the macro.
  • TelemetryHelpers: Buffer Agent now uses Agent.start_link (linked) with anonymous process naming instead of Agent.start (unlinked) with :global dynamic atom names, preventing resource leaks on test crashes.
  • ConcurrentHarness: Execution engine (~420 lines) extracted to ConcurrentHarness.Runtime, leaving the public module as a thin delegation layer.
  • ex_doc dependency bumped from ~> 0.27 to ~> 0.40.
  • Supertester moduledoc updated from "Multi-repository test orchestration" to "OTP-focused testing toolkit".
  • PerformanceHelpers.assert_no_memory_leak/3: Reworked to use warmup + evenly distributed sampling + robust trend analysis instead of a fragile first/last sample delta. Added optional tuning keys (:sample_count, :warmup_iterations, :min_absolute_growth_bytes, :min_upward_ratio).
  • SupervisorHelpers / ChaosHelpers / ProcessLifecycle: Consolidated supervisor child safety and restart-diff logic to shared SupervisorIntrospection internals.
  • ETSIsolation context setup paths: Reduced duplicated context wiring by using shared IsolationContextStore.put_updated/2.
  • ETSIsolation injection cleanup: Removed legacy cleanup entry-shape handling; cleanup now processes canonical {injection_id, injection, delete_on_cleanup?} entries only.
  • TelemetryHelpers.with_telemetry/3: Removed redundant buffer flush after handler detach.
  • SharedRegistry.ensure_started/0: Simplified to a global-lock guarded start-or-reuse path with targeted stale recovery; avoids multi-branch retry loops while keeping reset recovery behavior.
  • ChaosHelpers.inject_crash/3 ({:after_ms, _}): Delayed injectors now monitor the target and cancel immediately on target exit, enforcing guaranteed cancellation semantics.
  • PerformanceHelpers.measure_mailbox_growth/3: Mailbox monitor tasks are now always stopped/joined, including exception (raise/throw/exit) paths.
  • TelemetryMailbox + TelemetryHelpers selective flush paths: Consolidated flush filtering internals and switched selective flush behavior to preserve non-matching telemetry messages.
  • ETSIsolation.setup_ets_isolation/*: Setup overloads now share one initialization pipeline and explicit context fetch path (removed no-op context update pattern).

Fixed

  • ChaosHelpers.chaos_kill_children/2: No longer exits the caller when the supervisor dies mid-loop; returns a report with supervisor_crashed: true. Now accepts registered supervisor names (atom, {:global, _}, {:via, _, _}). Handles duplicate child IDs correctly (e.g., dynamic children with :undefined IDs).
  • GenServerHelpers.cast_and_sync/4: Recognizes missing sync handlers when the server crashes with a FunctionClauseError exit shape. Returns {:error, :missing_sync_handler} in non-strict mode when sync probing reveals a missing handler via process exit. Documentation now correctly distinguishes bare :ok return from {:ok, reply} return.
  • SupervisorHelpers.test_restart_strategy/3: No longer misclassifies removed temporary children as restarted. Enforces expected strategy/module checks including map-based supervisor internals (DynamicSupervisor).
  • Assertions.assert_no_process_leaks/1: Corrected :spawned trace handling; improved persistence checks to reduce false positives from unrelated/self trace events while preserving delayed-descendant leak detection.
  • TelemetryHelpers buffered event capture no longer creates per-process dynamic atoms for buffer naming. Buffer Agent no longer leaks when test process crashes (now linked to caller).
  • Isolation and helper naming paths no longer rely on unbounded dynamic atom creation (test_id is string-based; shared registry/process naming are atom-safe).
  • ProcessLifecycle.stop_process_safely/2 uses :shutdown instead of :normal exit reason and unlinks before stopping to prevent exit propagation to the test process.
  • ProcessRef.resolve/1 normalizes :undefined from :global.whereis_name and module.whereis_name to nil instead of leaking the :undefined atom.
  • UnifiedTestFoundation.supervisor_ready_probe/1 now rescues/catches exits instead of crashing the poller.
  • ETS injection restore is now idempotent — duplicate restore calls are tracked and skipped via injection IDs.
  • Shared registry is unlinked from creator to survive when spawned from short-lived Task processes.
  • Refactored GenServerHelpers.concurrent_calls/4 internals to satisfy strict Credo nesting limits.
  • Intermittent full-suite failures caused by shared registry startup races and stale PIDPartition processes after reset/teardown.
  • Intermittent false negatives in assert_no_memory_leak/3 under suite load by requiring sustained upward trend + meaningful absolute growth before flagging leaks.
  • MessageHarness.trace_messages/3 timeout path now raises a descriptive RuntimeError and shuts down the collector task deterministically.
  • TelemetryHelpers.flush_telemetry/1 no longer drops unrelated telemetry messages when flushing a specific event pattern.
  • Delayed crash injectors no longer linger for full timeout after target termination.
  • measure_mailbox_growth/3 no longer leaks monitor tasks when wrapped operations fail via exceptions or exits.

Documentation

  • README and guides refreshed for 0.6.0 API behavior (strict sync guidance, supervisor validation semantics, chaos suite timeout behavior, expanded helper coverage).
  • cast_and_sync/4 return value semantics corrected across README, QUICK_START, MANUAL, and API_GUIDE.
  • Restored onboarding content: prerequisites (TestableGenServer), common patterns, and troubleshooting sections added to QUICK_START and MANUAL.
  • Supertester.Env documented in API_GUIDE and MANUAL with behaviour spec and custom implementation example.
  • Added skip_code_autolink_to for Supertester.Env.ExUnit and Supertester.Internal.SupervisorIntrospection to fix docs build warnings.
  • Added migration guide for upgrading from 0.5.1 to 0.6.0.

Migration from 0.5.1

If upgrading from 0.5.x to 0.6.0, note the following breaking changes:

  1. cast_and_sync/4 missing-sync behavior: Non-strict mode now consistently returns {:error, :missing_sync_handler} instead of :ok for servers without sync handlers. Update assertions that matched bare :ok for servers without sync support.

  2. test_restart_strategy/3 raises on missing children and strategy mismatch: Scenario child IDs that do not exist in the supervisor now raise ArgumentError. The supervisor's actual strategy must also match the expected strategy. Ensure all scenario child IDs match actual supervisor children.

  3. assert_supervision_tree_structure/2 validates child modules: Leaf children now have their modules checked. Tests that only validated child IDs may need updating.

  4. ETSIsolation.inject_table/3-4 atom safety: Fallback path no longer creates dynamic atoms. Use __supertester_set_table__/2 callback or pre-existing env keys.

  5. Process naming format: Process names generated by setup_isolated_genserver/3 and setup_isolated_supervisor/3 use {:via, Registry, {:supertester_shared_registry, ...}} instead of {:global, ...} tuples. Code that pattern-matches on the exact name structure must be updated.

  6. generate_test_id/1 returns a string: Test IDs are now strings, not atoms. Code using is_atom(test_id) or Atom.to_string(test_id) must be updated to use to_string/1 or match on strings.

  7. safe_start/4 no longer sets trap_exit on the test process: The function now uses a spawned intermediary. Tests relying on the side effect of trap_exit being set must trap exits explicitly.

  8. wait_for_supervisor_stabilization/2 delegation changed: Now delegates to OTPHelpers.wait_for_supervisor_restart/2 rather than UnifiedTestFoundation.wait_for_supervision_tree_ready/2.

0.5.1 - 2026-01-09

Fixed

0.5.0 - 2026-01-06

Added

  • EchoLab example app demonstrating full Supertester coverage under examples/.

Changed

  • Documentation reorganized under guides/ and refreshed for 0.5.0 coverage.

Fixed

  • setup_isolated_supervisor/3 now passes init args correctly when starting supervisors.
  • GenServer helper name resolution now supports {:global, _} and {:via, _, _} registrations.
  • Mailbox monitoring handles process exits during sampling and avoids redundant messaging.
  • Telemetry buffering avoids Agent startup races under concurrency.
  • GenServer state assertions now support scalar expected values.
  • Supervisor strategy assertion documentation clarified to reflect public API limits.

Removed

  • get_supervisor_strategy/1 (unfinished API) has been dropped.
  • Legacy release summary documentation has been removed.

0.4.0 - 2025-12-26

Added

  • Supertester.TelemetryHelpers for async-safe telemetry assertions with test-scoped handlers, buffering, and helper macros.
  • Supertester.LoggerIsolation for per-process Logger level isolation and safe log capture helpers.
  • Supertester.ETSIsolation for isolated ETS table creation, mirroring, and table injection with automatic cleanup.
  • ExUnitFoundation options telemetry_isolation, logger_isolation, and ets_isolation, plus tag-based telemetry, logger, and ETS configuration.
  • New telemetry events covering isolation setup, handler lifecycle, and ETS table operations.

Changed

  • Supertester.IsolationContext extended with telemetry, logger, and ETS isolation state fields.

0.3.1 - 2025-11-19

Added

  • Supertester.ConcurrentHarness providing a scenario-based runner with thread scripts, invariants, structured event reporting, and optional mailbox metrics.
  • Supertester.PropertyHelpers exposing genserver_operation_sequence/2 and concurrent_scenario/1 generators that emit configs consumed by the concurrent harness or property-based tests.
  • Supertester.MessageHarness with trace_messages/3 for lightweight mailbox visibility during any operation.
  • Regression tests covering the new harness layer, message tracing utility, and property generators (when :stream_data is available).
  • Supertester.Telemetry centralized module plus global :telemetry dependency, enabling harness lifecycle, chaos, mailbox, and performance events out of the box.
  • ConcurrentHarness chaos helpers (chaos_kill_children/1, chaos_inject_crash/2) and run_with_performance/2 helper for composing richer scenarios.
  • ChaosHelpers.run_chaos_suite/3 can now consume full Supertester.ConcurrentHarness scenarios (via :concurrent entries), consolidating telemetry and reporting across chaos suites.

Changed

  • PerformanceHelpers.measure_mailbox_growth/3 now accepts options (sampling interval) and returns the wrapped operation result so higher-level APIs can forward diagnostics.
  • assert_mailbox_stable/2 forwards extra options to the new mailbox measurement signature, enabling finer-grained monitoring from tests and the concurrent harness.
  • ConcurrentHarness integrates optional chaos hooks, performance expectations, and emits telemetry events for scenario start/stop and diagnostics.

Fixed

  • Documentation now reflects the new concurrent harness and mailbox utilities, replacing stale references to planned modules that did not yet exist.

0.3.0 - 2025-11-19

Added

  • Supertester.ExUnitFoundation — drop-in ExUnit adapter that automatically wires Supertester.UnifiedTestFoundation isolation and sets async mode based on the selected strategy.
  • Supertester.Env and configurable environment behaviour, enabling custom harnesses to override how on-exit cleanup callbacks are registered.
  • Supertester.IsolationContext struct for consistent tracking of processes, ETS tables, cleanup callbacks, and contextual tags.
  • Dedicated tests covering the new Env abstraction, ExUnit adapter behaviour, and enhanced helpers.

Changed

  • Supertester.UnifiedTestFoundation now focuses purely on managing isolation contexts, exposes helpers to fetch/update those contexts, and registers cleanup via the new environment abstraction. The legacy macro now warns and delegates to the ExUnit adapter.
  • Supertester.OTPHelpers improved process tracking, derives deterministic names from the isolation context, traps exits to prevent linked process crashes from taking down tests, and returns richer error metadata when isolated processes fail to start.
  • Supertester.GenServerHelpers gained stricter cast_and_sync/4, structured concurrent_calls/4 output, configurable timeouts, parallel stress-test workers, and better logging around missing sync handlers.
  • Documentation (README, MANUAL, API guide, technical design doc, etc.) updated to reflect the new ExUnit adapter, environment abstraction, and enhanced helper APIs.

Fixed

  • Stress tests now aggregate worker metrics reliably and include execution duration, avoiding missed worker exits.
  • cast_and_sync/4 handles servers that raise when sync handlers are missing, providing clearer errors when strict?: true is enabled.

0.2.1 - 2025-10-17

Fixed

  • Fixed missing logo asset in hex.pm package (404 error on hexdocs)

0.2.0 - 2025-10-07

Added

  • Supertester.TestableGenServer - Automatic __supertester_sync__ handler injection for GenServers
    • Enables deterministic testing without Process.sleep/1
    • Works with use Supertester.TestableGenServer in any GenServer
    • Supports state return option for debugging
  • Supertester.SupervisorHelpers - Comprehensive supervision tree testing toolkit
    • test_restart_strategy/3 - Test one_for_one, one_for_all, rest_for_one strategies
    • assert_supervision_tree_structure/2 - Validate tree structure
    • trace_supervision_events/2 - Monitor supervisor events
    • wait_for_supervisor_stabilization/2 - Wait for all children ready
    • get_active_child_count/1 - Get active child count
  • Supertester.ChaosHelpers - Chaos engineering for OTP resilience testing
    • inject_crash/3 - Controlled crash injection (immediate, delayed, random)
    • chaos_kill_children/2 - Random child killing in supervision trees
    • simulate_resource_exhaustion/2 - Process/ETS/memory limit simulation
    • assert_chaos_resilient/4 - Verify system recovery from chaos
    • run_chaos_suite/3 - Comprehensive chaos scenario testing
  • Supertester.PerformanceHelpers - Performance testing and regression detection
    • assert_performance/2 - Assert time/memory/reduction bounds
    • assert_no_memory_leak/3 - Detect memory leaks over iterations
    • measure_operation/1 - Measure time, memory, and CPU work
    • measure_mailbox_growth/2 - Monitor mailbox growth
    • assert_mailbox_stable/2 - Ensure mailbox doesn't grow unbounded
    • compare_performance/1 - Compare multiple function performances

Changed

  • BREAKING: Eliminated all Process.sleep/1 calls in production code
    • Replaced with proper receive do after patterns
    • Modified OTPHelpers, GenServerHelpers, UnifiedTestFoundation, Assertions
    • All timing-based waits now use receive timeouts with proper remaining time calculation
  • Updated package description to emphasize chaos engineering and performance testing
  • Enhanced mix.exs with better hex.pm publishing metadata

Improved

  • All tests now run with 100% async execution
  • Test suite execution time: ~1.6 seconds for 37 tests
  • Zero compilation warnings in production code
  • Complete type specs on all public functions
  • Comprehensive documentation with real-world examples

Fixed

  • Pattern matching warnings in UnifiedTestFoundation and SupervisorHelpers
  • Unused variable warnings in test files
  • Proper handling of supervisor event tracing

Documentation

  • Added comprehensive technical design documentation.
  • Added implementation status tracking documentation.
  • Enhanced README.md with advanced usage examples for all new modules
  • Added chaos engineering, performance testing, and supervision tree examples

Testing

  • Added 33 new tests across 4 test files
  • Total test coverage: 37 tests, all passing
  • Added chaos engineering test scenarios
  • Added performance regression test examples
  • Added supervision tree verification tests

0.1.0 - 2025-07-22

Added

  • Initial release of Supertester
  • Multi-repository test orchestration and execution framework
  • OTP-compliant testing utilities to replace Process.sleep/1
  • Test isolation support enabling async: true
  • GenServer testing helpers with proper synchronization
  • Supervisor testing utilities for supervision tree validation
  • Performance testing framework for benchmarking and load testing
  • Chaos engineering helpers for resilience testing
  • Custom OTP-aware assertions
  • Unified test foundation with multiple isolation modes
  • Complete elimination of GenServer registration conflicts
  • Zero test failure guarantee across monorepo structures

Features