Submodule bindings for vllm.config.
Version
- Requested: 0.14.0
- Observed at generation: 0.14.0
Runtime Options
All functions accept a __runtime__ option for controlling execution behavior:
Vllm.Config.some_function(args, __runtime__: [timeout: 120_000])Supported runtime options
:timeout- Call timeout in milliseconds (default: 120,000ms / 2 minutes):timeout_profile- Use a named profile (:default,:ml_inference,:batch_job,:streaming):stream_timeout- Timeout for streaming operations (default: 1,800,000ms / 30 minutes):session_id- Override the session ID for this call:pool_name- Target a specific Snakepit pool (multi-pool setups):affinity- Override session affinity (:hint,:strict_queue,:strict_fail_fast)
Timeout Profiles
:default- 2 minute timeout for regular calls:ml_inference- 10 minute timeout for ML/LLM workloads:batch_job- Unlimited timeout for long-running jobs:streaming- 2 minute timeout, 30 minute stream_timeout
Example with timeout override
# For a long-running ML inference call
Vllm.Config.predict(data, __runtime__: [timeout_profile: :ml_inference])
# Or explicit timeout
Vllm.Config.predict(data, __runtime__: [timeout: 600_000])
# Route to a pool and enforce strict affinity
Vllm.Config.predict(data, __runtime__: [pool_name: :strict_pool, affinity: :strict_queue])See SnakeBridge.Defaults for global timeout configuration.
Summary
Functions
Python module attribute vllm.config.__all__.
A decorator that ensures all fields in a dataclass have default values
type(object) -> the object's type
Get any docstrings placed after attribute assignments in a class body.
Cache config to avoid repeated calls to get_current_vllm_config()
Python binding for vllm.config.get_current_vllm_config.
Python binding for vllm.config.get_current_vllm_config_or_none.
Get layers from the vLLM config.
Python binding for vllm.config.is_init_field.
Python binding for vllm.config.iter_architecture_defaults.
Temporarily set the current vLLM config.
Python binding for vllm.config.str_dtype_to_torch_dtype.
Python binding for vllm.config.try_match_architecture_defaults.
Python binding for vllm.config.update_config.
Functions
@spec __all__() :: {:ok, [term()]} | {:error, Snakepit.Error.t()}
Python module attribute vllm.config.__all__.
Returns
list(term())
@spec config( term(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
A decorator that ensures all fields in a dataclass have default values
and that each field has a docstring.
If a ConfigT is used as a CLI argument itself, the type keyword argument
provided by get_kwargs will be
pydantic.TypeAdapter(ConfigT).validate_json(cli_arg) which treats the
cli_arg as a JSON string which gets validated by pydantic.
Config validation is performed by the tools/pre_commit/validate_config.py script, which is invoked during the pre-commit checks.
Parameters
cls(term())
Returns
term()
@spec config_type(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
type(object) -> the object's type
type(name, bases, dict, **kwds) -> a new type
Parameters
args(term())kwargs(term())
Returns
term()
@spec get_attr_docs( term(), keyword() ) :: {:ok, %{optional(String.t()) => String.t()}} | {:error, Snakepit.Error.t()}
Get any docstrings placed after attribute assignments in a class body.
https://davidism.com/mit-license/
Parameters
cls(term())
Returns
%{optional(String.t()) => String.t()}
@spec get_cached_compilation_config(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Cache config to avoid repeated calls to get_current_vllm_config()
Returns
term()
@spec get_current_vllm_config(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.get_current_vllm_config.
Returns
term()
@spec get_current_vllm_config_or_none(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.get_current_vllm_config_or_none.
Returns
term()
@spec get_layers_from_vllm_config(term(), term()) :: {:ok, %{optional(String.t()) => term()}} | {:error, Snakepit.Error.t()}
Get layers from the vLLM config.
Parameters
vllm_config- The vLLM config.layer_type- The type of the layer to get.layer_names- The names of the layers to get. If None, return all layers.
Returns
%{optional(String.t()) => term()}
@spec is_init_field(term(), String.t(), keyword()) :: {:ok, boolean()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.is_init_field.
Parameters
cls(term())name(String.t())
Returns
boolean()
@spec iter_architecture_defaults(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.iter_architecture_defaults.
Returns
term()
@spec set_current_vllm_config(term()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Temporarily set the current vLLM config.
Used during model initialization. We save the current vLLM config in a global variable, so that all modules can access it, e.g. custom ops can access the vLLM config to determine how to dispatch.
Parameters
vllm_config(term())check_compile(term() default: False)prefix(term() default: None)
Returns
term()
@spec set_current_vllm_config( term(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec set_current_vllm_config(term(), term()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec set_current_vllm_config(term(), term(), keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec set_current_vllm_config(term(), term(), term()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec str_dtype_to_torch_dtype( String.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.str_dtype_to_torch_dtype.
Parameters
type(String.t())
Returns
term()
@spec try_match_architecture_defaults( String.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.try_match_architecture_defaults.
Parameters
architecture(String.t())runner_type(term() | nil keyword-only default: None)convert_type(term() | nil keyword-only default: None)
Returns
term()
@spec update_config(term(), %{optional(String.t()) => term()}, keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Python binding for vllm.config.update_config.
Parameters
config(term())overrides(%{optional(String.t()) => term()})
Returns
term()