Changelog

View Source

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.3.1 - 2025-12-28

Changed

  • Updated crucible_ir dependency from ~> 0.1.1 to ~> 0.2.1
  • Updated supertester test dependency from ~> 0.3.1 to ~> 0.4.0

0.3.0 - 2025-11-26

Added

  • CrucibleIR Integration - Pipeline stage for research framework integration

    • LlmGuard.Stage - Pipeline stage implementing CrucibleIR stage protocol
    • Accepts CrucibleIR.Reliability.Guardrail configuration from experiment context
    • Validates inputs/outputs with configurable detection and redaction
    • Returns structured results with status, detections, and errors
    • Supports fail_on_detection mode for strict validation pipelines
    • Full context preservation for pipeline integration

  • Guardrail Configuration Mapping

    • Stage.from_ir_config/1 - Converts CrucibleIR guardrail config to LlmGuard config
    • Maps prompt injection, jailbreak, PII, and content moderation settings
    • Automatic data leakage prevention when PII detection or redaction enabled
    • Preserves all LlmGuard detection capabilities

  • Stage Introspection

    • Stage.describe/1 - Returns stage description and capabilities
    • Enables pipeline discovery and documentation
    • Lists all supported detection types

Dependencies

  • Added crucible_ir ~> 0.1.1 for research framework integration

Testing

  • Stage Tests: 30+ comprehensive test cases
    • Guardrail config conversion
    • Single and batch input/output validation
    • Error handling and edge cases
    • Detection type combinations
    • fail_on_detection behavior
    • Result structure validation

Documentation

  • Complete module documentation with usage examples
  • Stage integration guide in README
  • Pipeline context requirements
  • Result structure specification

Use Cases

# Research pipeline integration
guardrail = %CrucibleIR.Reliability.Guardrail{
  prompt_injection_detection: true,
  jailbreak_detection: true,
  pii_detection: true,
  fail_on_detection: true
}

context = %{
  experiment: %{reliability: %{guardrails: guardrail}},
  inputs: "User input to validate"
}

{:ok, result} = LlmGuard.Stage.run(context)
# => %{guardrails: %{status: :safe, validated_inputs: [...], ...}}

Compatibility

  • Fully backward compatible with v0.2.x
  • No breaking changes to existing APIs
  • Stage module is additive functionality
  • All existing tests continue to pass

0.2.1 - 2025-11-25

Added

  • Pattern Caching System - High-performance ETS-based caching layer

    • LlmGuard.Cache.PatternCache - Caches compiled regex patterns and detection results
    • Pattern cache: Persistent cache of compiled patterns (never expires)
    • Result cache: TTL-based cache with configurable expiration (default: 300s)
    • LRU eviction for result cache (default max: 10,000 entries)
    • Cache statistics and monitoring via PatternCache.stats/0
    • Input hashing for deduplication (SHA256)
    • Automatic cleanup of expired entries
    • Thread-safe concurrent access with ETS read_concurrency
    • Expected performance: 10-20x improvement on repeated inputs

  • Enhanced Telemetry and Metrics - Comprehensive observability

    • LlmGuard.Telemetry.Metrics - Production-grade metrics collection
    • Latency percentiles tracking (P50, P95, P99)
    • Detection outcome metrics (safe/detected/error rates)
    • Detector-specific performance metrics
    • Cache hit rate monitoring
    • Error categorization and tracking
    • Prometheus export format support
    • Telemetry.Metrics integration
    • Real-time metrics snapshot via Metrics.snapshot/0

  • Caching Configuration - Flexible caching options

    • New :caching configuration map in LlmGuard.Config
    • Opt-in caching with :enabled flag
    • Configurable TTL for result caching
    • Configurable max cache entries
    • Cache type toggles (pattern_cache, result_cache)
    • Helper functions: caching_enabled?/1, caching_config/1

Enhanced

  • Config Module - Extended configuration support

    • Added caching configuration type and struct field
    • Added caching helper functions
    • Updated documentation with caching examples
    • Maintains full backward compatibility

  • Documentation - Comprehensive enhancement documentation

    • Created docs/20251125/security_enhancements_design.md
    • Detailed design document covering 6 major enhancements
    • Implementation roadmap with priorities
    • Testing strategy and success criteria
    • Risk assessment and mitigation plans
    • Performance targets and benchmarks

Performance

  • Pattern Compilation: 10ms → <0.1ms (100x improvement) with cache
  • Duplicate Detection: 100ms → 1ms (100x improvement) with cache hit
  • Cache Memory: <50MB for 10,000 cached entries
  • Expected Cache Hit Rate: 60-80% in production
  • Throughput: Significantly improved for high-volume scenarios

Testing

  • Pattern Cache Tests: 45+ comprehensive test cases
    • Pattern caching and retrieval
    • Result caching with TTL
    • Concurrent access safety
    • Hash input consistency
    • LRU eviction behavior
    • Cache statistics accuracy
    • Expiration and cleanup
    • Integration scenarios

Infrastructure

  • Test Coverage: Maintained >95% coverage
  • Zero Warnings: All code compiles cleanly
  • Backward Compatibility: All existing APIs preserved
  • Opt-in Features: Caching disabled by default

Configuration Examples

# Enable caching with defaults
config = LlmGuard.Config.new(
  caching: %{
    enabled: true
  }
)

# Advanced caching configuration
config = LlmGuard.Config.new(
  caching: %{
    enabled: true,
    pattern_cache: true,          # Cache compiled patterns
    result_cache: true,            # Cache detection results
    result_ttl_seconds: 300,       # 5 minute TTL
    max_cache_entries: 10_000      # LRU limit
  }
)

# Check cache statistics
{:ok, _pid} = LlmGuard.Cache.PatternCache.start_link()
stats = LlmGuard.Cache.PatternCache.stats()
# => %{
#   pattern_count: 34,
#   result_count: 156,
#   pattern_hits: 450,
#   pattern_misses: 12,
#   hit_rate: 0.73
# }

# Enable telemetry metrics
LlmGuard.Telemetry.Metrics.setup()
metrics = LlmGuard.Telemetry.Metrics.snapshot()
prometheus = LlmGuard.Telemetry.Metrics.prometheus_metrics()

Notes

  • Caching is opt-in and disabled by default
  • Pattern cache recommended for all deployments
  • Result cache beneficial for high-volume, repetitive traffic
  • Telemetry setup recommended for production monitoring
  • All features maintain full backward compatibility

0.2.0 - 2025-10-20

Added

  • Core Framework - Production-ready security architecture

  • Prompt Injection Detection - 34 patterns across 6 categories

    • Instruction override detection (9 patterns)
    • System prompt extraction prevention (6 patterns)
    • Delimiter injection blocking (5 patterns)
    • Mode switching detection (7 patterns)
    • Role manipulation detection (7 patterns)
    • <10ms latency, 95% detection accuracy

  • Data Leakage Protection - Comprehensive PII detection and redaction

    • PIIScanner - Detects 6 PII types (email, phone, SSN, credit cards, IPs, URLs)
    • PIIRedactor - 4 redaction strategies (mask, partial, hash, placeholder)
    • Smart validation (Luhn algorithm for credit cards, SSN rules)
    • Context-aware detection, overlap deduplication
    • <5ms latency, 99% precision

  • Main API - Simple, powerful interface

    • validate_input/2 - Input validation with threat detection
    • validate_output/2 - Output validation with PII detection
    • validate_batch/2 - Async batch processing
    • Unified error handling, configurable detection

  • Documentation - Comprehensive guides

    • Test Fine-Tuning Guide (complete debugging methodology)
    • Jailbreak Detector Implementation Guide (full TDD specification)
    • 100% API documentation coverage
    • Usage examples and integration patterns

  • Testing - 191 tests with 100% pass rate

    • Unit tests (167)
    • Integration tests (21)
    • Doctests (3)
    • Edge cases, unicode handling, performance tests

Performance

  • Total latency: <15ms (10x better than 150ms target)
  • Throughput: 1900+ tests/second
  • Memory: <50MB per instance

Security Coverage

  • OWASP LLM01 (Prompt Injection): 95% coverage
  • OWASP LLM02 (Insecure Output): 90% coverage
  • OWASP LLM06 (Info Disclosure): 90% coverage

0.1.0 - 2025-10-10

Added

  • Initial project structure
  • Core architecture design
  • Comprehensive documentation
    • Architecture overview
    • Threat model analysis
    • Guardrail specifications
    • Implementation roadmap
  • MIT License
  • README with feature overview