TantivyEx.ResourceManager (TantivyEx v0.4.1)

View Source

Comprehensive resource management and lifecycle control for TantivyEx.

This module provides proper resource cleanup, lifecycle management, and resource pooling to ensure efficient resource usage and prevent resource leaks in TantivyEx operations.

Features

  • Resource Lifecycle Management: Automatic tracking and cleanup
  • Resource Pooling: Efficient reuse of expensive resources
  • Leak Detection: Detection and prevention of resource leaks
  • Graceful Shutdown: Proper cleanup during system shutdown
  • Resource Monitoring: Real-time resource usage tracking
  • Automatic Cleanup: Cleanup based on usage patterns and memory pressure

Resource Types

The system manages several types of resources:

  • Index Resources: Index instances and their associated files
  • Writer Resources: Index writers with their buffer management
  • Reader Resources: Index readers and searchers
  • Query Resources: Query parsers and compiled queries
  • Native Resources: Native Rust objects and memory allocations
  • Cache Resources: Various caches and temporary data

Configuration

TantivyEx.ResourceManager.configure(%{
  # Resource limits
  max_writers: 10,
  max_readers: 50,
  max_cached_queries: 1000,

  # Cleanup policies
  cleanup_interval_ms: 30_000,
  idle_timeout_ms: 300_000,     # 5 minutes
  force_cleanup_threshold: 0.9, # 90% resource usage

  # Pool configuration
  enable_pooling: true,
  pool_min_size: 2,
  pool_max_size: 10,
  pool_checkout_timeout_ms: 5000,

  # Monitoring
  track_usage: true,
  leak_detection: true,
  usage_logging: true
})

Usage

# Start the resource manager
{:ok, _pid} = TantivyEx.ResourceManager.start_link()

# Register resources for management
{:ok, writer} = TantivyEx.IndexWriter.new(index, 50_000_000)
:ok = TantivyEx.ResourceManager.register(writer, :writer, &IndexWriter.close/1)

# Use pooled resources
TantivyEx.ResourceManager.with_pooled_reader(index, fn reader ->
  TantivyEx.Searcher.search(reader, query, 10)
end)

# Manual resource management
{:ok, stats} = TantivyEx.ResourceManager.get_resource_stats()
:ok = TantivyEx.ResourceManager.cleanup_idle_resources()
:ok = TantivyEx.ResourceManager.force_cleanup()

Summary

Types

pool_config()
resource_info()
resource_stats()
resource_type()

Functions

checkin(pool_id, resource)

Checks in a resource to a pool.

checkout(pool_id, timeout \\ 5000)

Checks out a resource from a pool.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

cleanup_idle_resources()

Forces cleanup of all idle resources.

cleanup_resource(resource_id)

Cleans up a specific resource by ID.

configure(config)

Configures the resource manager.

create_pool(pool_id, type, factory_fun, config \\ %{})

Creates a resource pool for a specific resource type.

destroy_pool(pool_id)

Destroys a resource pool.

detect_leaks()

Detects potential resource leaks.

force_cleanup()

Forces cleanup of all resources (emergency cleanup).

get_resource(resource_id)

Gets a specific resource by ID.

get_resource_stats()

Gets current resource statistics.

list_resources()

Lists all registered resources.

register(resource, type, cleanup_fun, metadata \\ %{})

Registers a resource for management.

register_resource(resource_id, resource_data)

Registers a resource with simplified API for tests.

start_link(opts \\ [])

Starts the resource manager.

unregister(resource)

Unregisters a resource from management.

unregister_resource(resource_id)

Unregisters a resource by ID.

with_pooled_resource(pool_id, fun, timeout \\ 5000)

Executes a function with a pooled resource.

Types

pool_config()

@type pool_config() :: %{
  min_size: non_neg_integer(),
  max_size: non_neg_integer(),
  checkout_timeout_ms: non_neg_integer(),
  idle_timeout_ms: non_neg_integer()
}

resource_info()

@type resource_info() :: %{
  resource: any(),
  type: resource_type(),
  cleanup_fun: function(),
  created_at: DateTime.t(),
  last_used: DateTime.t(),
  usage_count: non_neg_integer(),
  pool_id: String.t() | nil,
  metadata: map()
}

resource_stats()

@type resource_stats() :: %{
  total_resources: non_neg_integer(),
  by_type: map(),
  pool_stats: map(),
  cleanup_stats: map(),
  memory_usage: map()
}

resource_type()

@type resource_type() :: :writer | :reader | :index | :query | :native | :cache

Functions

checkin(pool_id, resource)

@spec checkin(String.t(), any()) :: :ok | {:error, TantivyEx.Error.t()}

Checks in a resource to a pool.

checkout(pool_id, timeout \\ 5000)

@spec checkout(String.t(), non_neg_integer()) ::
  {:ok, any()} | {:error, :timeout | :pool_not_found | TantivyEx.Error.t()}

Checks out a resource from a pool.

Parameters

  • pool_id - Pool identifier
  • timeout - Checkout timeout in milliseconds

Returns

  • {:ok, resource} - Successfully checked out resource
  • {:error, :timeout} - Checkout timed out
  • {:error, :pool_not_found} - Pool doesn't exist

Examples

iex> {:ok, reader} = TantivyEx.ResourceManager.checkout("readers", 5000)
iex> # Use reader
iex> :ok = TantivyEx.ResourceManager.checkin("readers", reader)

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

cleanup_idle_resources()

@spec cleanup_idle_resources() :: :ok | {:error, TantivyEx.Error.t()}

Forces cleanup of all idle resources.

cleanup_resource(resource_id)

@spec cleanup_resource(String.t()) :: :ok | {:error, :not_found | TantivyEx.Error.t()}

Cleans up a specific resource by ID.

configure(config)

@spec configure(map()) :: :ok | {:error, TantivyEx.Error.t()}

Configures the resource manager.

create_pool(pool_id, type, factory_fun, config \\ %{})

@spec create_pool(String.t(), resource_type(), function(), pool_config()) ::
  :ok | {:error, TantivyEx.Error.t()}

Creates a resource pool for a specific resource type.

Parameters

  • pool_id - Unique identifier for the pool
  • type - Resource type
  • factory_fun - Function to create new resources
  • config - Pool configuration

Examples

iex> TantivyEx.ResourceManager.create_pool("readers", :reader,
...>   fn -> TantivyEx.IndexReader.new(index) end,
...>   %{min_size: 2, max_size: 10}
...> )
:ok

destroy_pool(pool_id)

@spec destroy_pool(String.t()) :: :ok | {:error, TantivyEx.Error.t()}

Destroys a resource pool.

detect_leaks()

@spec detect_leaks() :: {:ok, [map()]} | {:error, TantivyEx.Error.t()}

Detects potential resource leaks.

Returns

Returns a list of resources that may be leaked based on usage patterns.

force_cleanup()

@spec force_cleanup() :: :ok | {:error, TantivyEx.Error.t()}

Forces cleanup of all resources (emergency cleanup).

get_resource(resource_id)

@spec get_resource(String.t()) :: any() | nil

Gets a specific resource by ID.

get_resource_stats()

@spec get_resource_stats() :: {:ok, resource_stats()} | {:error, TantivyEx.Error.t()}

Gets current resource statistics.

list_resources()

@spec list_resources() :: list()

Lists all registered resources.

register(resource, type, cleanup_fun, metadata \\ %{})

@spec register(any(), resource_type(), function(), map()) ::
  :ok | {:error, TantivyEx.Error.t()}

Registers a resource for management.

Parameters

  • resource - The resource to manage
  • type - The resource type
  • cleanup_fun - Function to call for cleanup
  • metadata - Optional metadata

Examples

iex> TantivyEx.ResourceManager.register(writer, :writer, &IndexWriter.close/1)
:ok

register_resource(resource_id, resource_data)

@spec register_resource(String.t(), map()) :: :ok | {:error, TantivyEx.Error.t()}

Registers a resource with simplified API for tests.

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

Starts the resource manager.

unregister(resource)

@spec unregister(any()) :: :ok

Unregisters a resource from management.

unregister_resource(resource_id)

@spec unregister_resource(String.t()) :: :ok | {:error, TantivyEx.Error.t()}

Unregisters a resource by ID.

with_pooled_resource(pool_id, fun, timeout \\ 5000)

@spec with_pooled_resource(String.t(), function(), non_neg_integer()) ::
  {:ok, any()} | {:error, TantivyEx.Error.t()}

Executes a function with a pooled resource.

The resource is automatically checked out before the function execution and checked back in afterward, even if an error occurs.

Parameters

  • pool_id - Pool identifier
  • fun - Function to execute with the resource
  • timeout - Checkout timeout

Examples

iex> TantivyEx.ResourceManager.with_pooled_resource("readers", fn reader ->
...>   TantivyEx.Searcher.search(reader, query, 10)
...> end)
{:ok, search_results}