Enterprise-grade multi-layer caching system for VaultX.

⚠️ Experimental Feature: This caching system is currently experimental and may undergo breaking changes in future versions. Use with caution in production environments.

📊 Coverage Note: This module is excluded from test coverage requirements as it's an experimental feature undergoing rapid development. See .coveralls.exs for details.

This module provides a sophisticated caching infrastructure with three distinct layers optimized for different access patterns and durability requirements. The system is designed for high-performance Vault operations with intelligent data management.

Architecture Overview

The cache system implements a hierarchical storage architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                        │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│  L1 Cache (Memory)  │  ETS Tables, ~1μs latency            │
│  • Ultra-fast access                                       │
│  • LRU eviction                                            │
│  • Configurable size limits                               │
└─────────────────────┬───────────────────────────────────────┘
                      │ (on miss)
┌─────────────────────▼───────────────────────────────────────┐
│  L2 Cache (Distributed) │  Pluggable adapters, ~10μs      │
│  • Cross-node sharing                                      │
│  • Memory/Redis adapters                                   │
│  • Automatic promotion to L1                              │
└─────────────────────┬───────────────────────────────────────┘
                      │ (on miss)
┌─────────────────────▼───────────────────────────────────────┐
│  L3 Cache (Persistent) │  File system, ~10ms latency      │
│  • Survives restarts                                       │
│  • Optional AES-256-GCM encryption                         │
│  • Automatic promotion to L1/L2                           │
└─────────────────────┬───────────────────────────────────────┘
                      │ (on miss)
┌─────────────────────▼───────────────────────────────────────┐
│              Original Data Source (Vault)                  │
└─────────────────────────────────────────────────────────────┘

Key Features

Performance Optimization

• Hierarchical Access: Fastest cache checked first
• Automatic Promotion: Frequently accessed data moves to faster layers
• Intelligent Eviction: LRU and TTL-based cleanup
• Concurrent Operations: Lock-free reads, safe writes
• Memory Management: Configurable limits with automatic cleanup

Reliability & Durability

• Graceful Degradation: System continues if individual layers fail
• Atomic Operations: Consistent state during failures
• Process Monitoring: Automatic restart of failed cache layers
• Data Integrity: Checksums and validation for persistent storage
• Backup & Recovery: Export/import capabilities for cache data

Security & Compliance

• Encryption at Rest: AES-256-GCM for L3 persistent storage
• Secure Key Management: Multiple key sources with proper permissions
• Access Control: Integration with VaultX authentication
• Audit Logging: Comprehensive operation tracking
• Data Classification: Support for sensitive data handling

Usage Examples

# Basic cache operations
{:ok, secret} = Vaultx.Cache.get("secret/myapp/config")
:ok = Vaultx.Cache.put("secret/myapp/config", secret_data)
:ok = Vaultx.Cache.delete("secret/myapp/config")

# TTL-based caching
:ok = Vaultx.Cache.put("temp_token", token, ttl: :timer.minutes(5))

# Get-or-compute pattern (recommended)
secret = Vaultx.Cache.get_or_compute("secret/expensive/operation", fn ->
  # This function only runs on cache miss
  fetch_expensive_secret_from_vault()
end, ttl: :timer.hours(1))

# Bulk operations
{:ok, results} = Vaultx.Cache.get_many(["key1", "key2", "key3"])
:ok = Vaultx.Cache.put_many([{"key1", "val1"}, {"key2", "val2"}])

# Pattern-based operations
:ok = Vaultx.Cache.clear("secret/myapp/*")  # Clear specific patterns
:ok = Vaultx.Cache.clear(:all)              # Clear all caches

# Cache warming (preload frequently accessed data)
:ok = Vaultx.Cache.warm("secret/myapp/*", &load_secret_function/1)

# Monitoring and statistics
{:ok, stats} = Vaultx.Cache.stats()
# Returns comprehensive statistics for all cache layers

Configuration

# Application configuration
config :vaultx,
  # Core cache settings
  cache_enabled: true,
  cache_eviction_policy: :lru,
  cache_max_memory_usage: 100 * 1024 * 1024,  # 100MB

  # L1 Memory Cache
  cache_l1_enabled: true,
  cache_l1_max_size: 10_000,
  cache_l1_ttl_default: 900_000,  # 15 minutes
  cache_l1_cleanup_interval: 300_000,  # 5 minutes

  # L2 Distributed Cache
  cache_l2_enabled: true,
  cache_l2_adapter: Vaultx.Cache.Adapters.Memory,
  cache_l2_max_size: 50_000,
  cache_l2_ttl_default: 3_600_000,  # 1 hour
  cache_l2_cleanup_interval: 600_000,  # 10 minutes

  # L3 Persistent Cache
  cache_l3_enabled: false,
  cache_l3_storage_path: "/var/cache/vaultx",
  cache_l3_ttl_default: 86_400_000,  # 24 hours
  cache_l3_cleanup_interval: 3_600_000,  # 1 hour
  cache_l3_encryption: false,

  # Advanced features
  cache_warming_enabled: true,
  cache_metrics_enabled: true,
  cache_manager_cleanup_interval: 300_000  # 5 minutes

# Environment variable overrides
export VAULTX_CACHE_ENABLED=true
export VAULTX_CACHE_L3_ENABLED=true
export VAULTX_CACHE_L3_ENCRYPTION=true
export VAULTX_L3_ENCRYPTION_KEY="$(openssl rand -base64 32)"

Performance Characteristics

Integration with VaultX

Current Support Status

| KV v2 Read | ✅ Implemented | kv2:{mount}:{path}|{version} | 15 minutes | | KV v1 Read | ❌ Not Supported | - | - | | Auth Token Validation | 🚧 Planned | auth:token:{hash} | Token TTL | | Policy Evaluation | 🚧 Planned | policy:{name}:{hash} | 1 hour | | Mount Metadata | 🚧 Planned | mount:{path}:metadata | 1 hour | | Lease Information | 🚧 Planned | lease:{id} | Lease TTL |

Supported Operations

• KV v2 Secrets: Automatic caching of secret reads with intelligent TTL
  • Cache keys include mount path, secret path, and version
  • Automatic cache invalidation on write/delete operations
  • Configurable TTL with 15-minute default
  • Support for both versioned and latest reads

Planned Integrations

• Auth Tokens: Token validation caching with security considerations
• Policy Data: Policy evaluation caching for performance optimization
• Metadata: Mount and engine metadata caching for reduced API calls
• Leases: Lease information caching with proper expiration handling

Best Practices

Development

• Use get_or_compute/3 for expensive operations
• Set appropriate TTLs based on data sensitivity and change frequency
• Monitor cache hit rates and adjust configuration accordingly
• Test cache behavior in failure scenarios

Production

• Enable L3 persistent cache for critical data
• Use encryption for sensitive cached data
• Monitor memory usage and set appropriate limits
• Implement cache warming for frequently accessed data
• Set up proper monitoring and alerting

Security

• Enable encryption for sensitive data in L3 cache
• Use environment variables for encryption keys
• Regularly rotate encryption keys
• Monitor cache access patterns for anomalies
• Implement proper cache invalidation on security events