mix req_llm.model_compat (ReqLLM v1.0.0-rc.8)

View Source

Validate ReqLLM model coverage using the fixture system.

Models are sourced from priv/models_dev/*.json (synced via mix req_llm.model_sync). Fixture validation state is tracked in priv/supported_models.json (auto-generated).

Selection Principles

Models are selected using clear precedence: spec → type → sample

  • spec: Pattern over providers/models
    • When no spec provided (just mix mc): Uses default sets from config (:test_models or :test_embedding_models)
    • When spec provided (e.g., "anthropic:*", "*:*"): Uses ALL matching models from registry
  • type: Filters by operation capability using registry metadata
    • text (default): Only text-generation models
    • embedding: Only embedding models
    • all: Both text and embedding models
  • sample (optional): Further reduces using :sample_text_models or :sample_embedding_models. If not configured, falls back to one model per provider.

Important:

  • Only implemented providers are included (registry models without implementation are skipped)
  • Config lists (:test_models, :test_embedding_models) are defaults only, not hard filters
  • Explicit specs like "anthropic:*" test ALL registry models for that provider

Usage

mix req_llm.model_compat                    # Show covered models (passing fixtures)
mix req_llm.model_compat --sample           # Test sample models from config
mix req_llm.model_compat --available        # List all registry models (unfiltered)

### Test using local fixtures
mix req_llm.model_compat "anthropic:*"      # ALL Anthropic text models from registry
mix req_llm.model_compat "openai:gpt-4o"    # Specific model
mix req_llm.model_compat "*:*"              # ALL models from implemented providers

### Test by operation type
mix req_llm.model_compat "google:*" --type all        # Google text + embedding models
mix req_llm.model_compat "google:*" --type embedding  # Google embedding models only
mix req_llm.model_compat "*:*" --type text            # All implemented text models

### Sample subset testing
mix req_llm.model_compat --sample           # Sample subset (~1 per provider if not configured)
mix req_llm.model_compat "anthropic:*" --sample --type text

### Record new fixtures
mix req_llm.model_compat "openai:*" --record
mix req_llm.model_compat "google:*" --type embedding --record

Flags

--available        List all models from models.dev API registry (no implementation filter)
--sample           Further reduce to sample subset (see :sample_* config or fallback)
--type TYPE        Operation type: text (default), embedding, or all
--record           Re-record fixtures (live API calls)
--record-all       Force re-record all fixtures (ignores state)
--debug            Enable verbose fixture debugging

Notes

  • When no spec is provided (or "*:*" is used), only implemented providers are considered
  • If a spec refers to an unimplemented provider, it will be skipped with a warning
  • The final model list is deterministic and stable