TantivyEx.Memory (TantivyEx v0.4.1)

View Source

Comprehensive memory management and monitoring for TantivyEx.

This module provides memory limits, monitoring, and automatic cleanup mechanisms to ensure efficient resource usage and prevent out-of-memory conditions during indexing and search operations.

Features

  • Memory Limits: Configurable memory limits for operations
  • Memory Monitoring: Real-time memory usage tracking
  • Automatic Cleanup: Automatic resource cleanup mechanisms
  • Memory Pressure Detection: Early warning for memory pressure
  • Garbage Collection Integration: Smart GC triggering
  • Resource Pooling: Memory-aware resource pooling

Configuration

# Configure global memory limits
TantivyEx.Memory.configure(%{
  max_memory_mb: 1024,              # Total memory limit
  writer_memory_mb: 512,            # Memory limit for index writers
  search_memory_mb: 256,            # Memory limit for search operations
  aggregation_memory_mb: 128,       # Memory limit for aggregations
  gc_threshold: 0.8,                # GC trigger threshold (80%)
  monitoring_interval_ms: 5000,     # Memory monitoring interval
  cleanup_on_pressure: true         # Auto-cleanup on memory pressure
})

Usage

# Create memory-aware operations
{:ok, writer} = TantivyEx.IndexWriter.new(index, memory_limit: 256)

# Monitor memory during operations
TantivyEx.Memory.with_monitoring fn ->
  # Perform memory-intensive operations
  TantivyEx.Document.add_batch(writer, documents, schema)
end

# Manual memory management
{:ok, stats} = TantivyEx.Memory.get_stats()
:ok = TantivyEx.Memory.force_cleanup()
:ok = TantivyEx.Memory.trigger_gc()

Automatic Memory Management

The module provides automatic memory management through:

  1. Memory Monitoring: Continuous monitoring of memory usage
  2. Pressure Detection: Early detection of memory pressure conditions
  3. Automatic Cleanup: Triggered cleanup when thresholds are exceeded
  4. Smart GC: Intelligent garbage collection timing
  5. Resource Limiting: Automatic enforcement of memory limits

Memory Stats

The module tracks comprehensive memory statistics:

  • Current memory usage by component
  • Peak memory usage
  • Memory pressure events
  • GC statistics
  • Resource cleanup events

Summary

Types

memory_config()
memory_stats()

Functions

can_proceed?(operation, estimated_memory_mb)

Checks if an operation can proceed given current memory constraints.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

configure(config)

Configures the memory management system.

force_cleanup()

Forces immediate memory cleanup.

get_stats()

Gets current memory statistics.

register_resource(resource, cleanup_fun, category)

Registers a resource for automatic cleanup.

start_link(opts \\ [])

Starts the memory management system.

trigger_gc()

Triggers garbage collection.

under_pressure?()

Checks if the system is under memory pressure.

unregister_resource(resource)

Unregisters a resource from automatic cleanup.

with_monitoring(fun, opts \\ [])

Executes a function with memory monitoring.

Types

memory_config()

@type memory_config() :: %{
  max_memory_mb: non_neg_integer(),
  writer_memory_mb: non_neg_integer(),
  search_memory_mb: non_neg_integer(),
  aggregation_memory_mb: non_neg_integer(),
  gc_threshold: float(),
  monitoring_interval_ms: non_neg_integer(),
  cleanup_on_pressure: boolean(),
  auto_gc: boolean(),
  pressure_threshold: float()
}

memory_stats()

@type memory_stats() :: %{
  total_used_mb: float(),
  total_limit_mb: non_neg_integer(),
  writer_used_mb: float(),
  search_used_mb: float(),
  aggregation_used_mb: float(),
  system_used_mb: float(),
  gc_count: non_neg_integer(),
  cleanup_count: non_neg_integer(),
  pressure_events: non_neg_integer(),
  last_gc: DateTime.t() | nil,
  last_cleanup: DateTime.t() | nil
}

Functions

can_proceed?(operation, estimated_memory_mb)

@spec can_proceed?(atom(), non_neg_integer()) ::
  {:ok, boolean()} | {:error, TantivyEx.Error.t()}

Checks if an operation can proceed given current memory constraints.

Parameters

  • operation - The type of operation (:indexing, :search, :aggregation)
  • estimated_memory_mb - Estimated memory required for the operation

Examples

iex> TantivyEx.Memory.can_proceed?(:indexing, 100)
{:ok, true}

iex> TantivyEx.Memory.can_proceed?(:search, 1000)
{:error, %TantivyEx.Error.MemoryError{}}

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

configure(config)

@spec configure(memory_config()) :: :ok | {:error, TantivyEx.Error.t()}

Configures the memory management system.

Parameters

  • config - Memory configuration map (see module documentation)

Examples

iex> TantivyEx.Memory.configure(%{max_memory_mb: 2048})
:ok

force_cleanup()

@spec force_cleanup() :: :ok | {:error, TantivyEx.Error.t()}

Forces immediate memory cleanup.

This triggers immediate cleanup of unused resources and may trigger garbage collection if configured.

Examples

iex> TantivyEx.Memory.force_cleanup()
:ok

get_stats()

@spec get_stats() :: {:ok, memory_stats()} | {:error, TantivyEx.Error.t()}

Gets current memory statistics.

Returns

Returns detailed memory usage statistics including:

  • Current usage by component
  • Memory limits
  • GC and cleanup counts
  • Memory pressure events

Examples

iex> {:ok, stats} = TantivyEx.Memory.get_stats()
iex> stats.total_used_mb
245.6

register_resource(resource, cleanup_fun, category)

@spec register_resource(any(), function(), atom()) :: :ok

Registers a resource for automatic cleanup.

Resources registered with this function will be automatically cleaned up when memory pressure is detected or when the system shuts down.

Parameters

  • resource - The resource reference or identifier
  • cleanup_fun - Function to call for cleanup
  • category - Resource category (:writer, :reader, :cache, etc.)

Examples

iex> TantivyEx.Memory.register_resource(writer_ref, &IndexWriter.close/1, :writer)
:ok

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

Starts the memory management system.

Options

  • :name - Process name (default: TantivyEx.Memory)
  • :config - Memory configuration (merged with defaults)

trigger_gc()

@spec trigger_gc() :: :ok

Triggers garbage collection.

Examples

iex> TantivyEx.Memory.trigger_gc()
:ok

under_pressure?()

@spec under_pressure?() :: boolean()

Checks if the system is under memory pressure.

Returns

  • true - System is under memory pressure
  • false - Memory usage is normal

Examples

iex> TantivyEx.Memory.under_pressure?()
false

unregister_resource(resource)

@spec unregister_resource(any()) :: :ok

Unregisters a resource from automatic cleanup.

Examples

iex> TantivyEx.Memory.unregister_resource(writer_ref)
:ok

with_monitoring(fun, opts \\ [])

@spec with_monitoring(
  function(),
  keyword()
) :: {:ok, any()} | {:error, TantivyEx.Error.t()}

Executes a function with memory monitoring.

The function is executed with active memory monitoring, and cleanup is performed automatically if memory pressure is detected.

Parameters

  • fun - Function to execute
  • opts - Options (:operation_type, :memory_limit_mb)

Examples

iex> TantivyEx.Memory.with_monitoring(fn ->
...>   # Memory-intensive operation
...>   :result
...> end, operation_type: :indexing)
{:ok, :result}