API Reference VLLM v#0.3.0

SystemEnv(torch_version, is_debug_build, cuda_compiled_version, gcc_version, clang_version, cmake_version, os, libc_version, python_version, python_platform, is_cuda_available, cuda_runtime_version, cuda_module_loading, nvidia_driver_version, nvidia_gpu_models, cudnn_version, pip_version, pip_packages, conda_packages, hip_compiled_version, hip_runtime_version, miopen_runtime_version, caching_allocator_config, is_xnnpack_available, cpu_info, rocm_version, vllm_version, vllm_build_flags, gpu_topo, env_vars)

Vllm.Compilation

Submodule bindings for vllm.compilation.

Vllm.Config

Submodule bindings for vllm.config.

Vllm.Config.AttentionConfig

Configuration for attention mechanisms in vLLM.

Vllm.Config.CUDAGraphMode

Constants for the cudagraph mode in CompilationConfig.

Vllm.Config.CacheConfig

Configuration for the KV cache.

Vllm.Config.CompilationConfig

Configuration for compilation.

Vllm.Config.CompilationMode

The compilation approach used for torch.compile-based compilation of the

Vllm.Config.DeviceConfig

Configuration for the device to use for vLLM execution.

Vllm.Config.ECTransferConfig

Configuration for distributed EC cache transfer.

Vllm.Config.EPLBConfig

Configuration for Expert Parallel Load Balancing (EP).

Vllm.Config.KVEventsConfig

Configuration for KV event publishing.

Vllm.Config.KVTransferConfig

Configuration for distributed KV cache transfer.

Vllm.Config.LoRAConfig

Configuration for LoRA.

Vllm.Config.LoadConfig

Configuration for loading the model weights.

Vllm.Config.ModelConfig

Configuration for the model.

Vllm.Config.MultiModalConfig

Controls the behavior of multimodal models.

Vllm.Config.ObservabilityConfig

Configuration for observability - metrics and tracing.

Vllm.Config.ParallelConfig

Configuration for the distributed execution.

Vllm.Config.PassConfig

Configuration for custom Inductor passes.

Vllm.Config.PoolerConfig

Controls the behavior of output pooling in pooling models.

Vllm.Config.ProfilerConfig

Dataclass which contains profiler config for the engine.

Vllm.Config.SchedulerConfig

Scheduler configuration.

Vllm.Config.SpeculativeConfig

Configuration for speculative decoding.

Vllm.Config.SpeechToTextConfig

Configuration for speech-to-text models.

Vllm.Config.StructuredOutputsConfig

Dataclass which contains structured outputs config for the engine.

Vllm.Config.SupportsMetricsInfo

Wrapper for Python class SupportsMetricsInfo.

Vllm.Config.VllmConfig

Dataclass which contains all vllm-related configuration. This

Vllm.Connections

Submodule bindings for vllm.connections.

Vllm.Connections.HTTPConnection

Helper class to send HTTP requests.

Vllm.DeviceAllocator

Submodule bindings for vllm.device_allocator.

Vllm.Distributed

Submodule bindings for vllm.distributed.

Vllm.Distributed.DeviceCommunicatorBase

Base class for device-specific communicator.

Vllm.Distributed.GraphCaptureContext

GraphCaptureContext(stream: torch.cuda.streams.Stream)

Vllm.Distributed.GroupCoordinator

PyTorch ProcessGroup wrapper for a group of processes.

Vllm.Distributed.StatelessProcessGroup

A dataclass to hold a metadata store, and the rank, world_size of the

Vllm.Distributed.TensorMetadata

TensorMetadata(device, dtype, size)

Vllm.Engine

Submodule bindings for vllm.engine.

Vllm.Entrypoints

Submodule bindings for vllm.entrypoints.

Vllm.EnvOverride

Submodule bindings for vllm.env_override.

Vllm.Envs

Submodule bindings for vllm.envs.

Vllm.Exceptions

Custom exceptions for vLLM.

Vllm.Exceptions.VLLMValidationError

vLLM-specific validation error for request validation failures.

Vllm.ForwardContext

ForwardContext(no_compile_layers: dict[str, typing.Any], attn_metadata: dict[str, vllm.v1.attention.backend.AttentionMetadata] | list[dict[str, vllm.v1.attention.backend.AttentionMetadata]], virtual_engine: int, dp_metadata: vllm.forward_context.DPMetadata | None = None, cudagraph_runtime_mode: vllm.config.compilation.CUDAGraphMode = <CUDAGraphMode.NONE: 0>, batch_descriptor: vllm.forward_context.BatchDescriptor | None = None, ubatch_slices: list[vllm.v1.worker.ubatch_utils.UBatchSlice] | None = None, additional_kwargs: dict[str, typing.Any] = <factory>)

Vllm.ForwardContext.AttentionMetadata

Wrapper for Python class AttentionMetadata.

Vllm.ForwardContext.BatchDescriptor

Batch descriptor for cudagraph dispatching. We should keep the num of

Vllm.ForwardContext.DPMetadata

DPMetadata(max_tokens_across_dp_cpu: torch.Tensor, num_tokens_across_dp_cpu: torch.Tensor, local_sizes: list[int] | None = None)

Vllm.ForwardContext.Module

Submodule bindings for vllm.forward_context.

Vllm.Grpc

vLLM gRPC protocol definitions.

Vllm.Inputs

Submodule bindings for vllm.inputs.

Vllm.Inputs.DataPrompt

Represents generic inputs handled by IO processor plugins.

Vllm.Inputs.EmbedsInputs

Represents embeddings-based inputs.

Vllm.Inputs.EmbedsPrompt

Schema for a prompt provided via token embeddings.

Vllm.Inputs.EncoderDecoderInputs

The inputs in [LLMEngine][vllm.engine.llm_engine.LLMEngine] before they

Vllm.Inputs.ExplicitEncoderDecoderPrompt

Represents an encoder/decoder model input prompt,

Vllm.Inputs.TextPrompt

Schema for a text prompt.

Vllm.Inputs.TokenInputs

Represents token-based inputs.

Vllm.Inputs.TokensPrompt

Schema for a tokenized prompt.

Vllm.LLM

An LLM for generating texts from given prompts and sampling parameters.

Vllm.LLMEngine

Legacy LLMEngine for backwards compatibility.

Vllm.Logger

Logging configuration for vLLM.

Vllm.Logger.ColoredFormatter

Wrapper for Python class ColoredFormatter.

Vllm.Logger.NewLineFormatter

Wrapper for Python class NewLineFormatter.

Vllm.Logger.VllmLogger

Note

Vllm.LoggingUtils

Submodule bindings for vllm.logging_utils.

Vllm.LoggingUtils.ColoredFormatter

Adds ANSI color codes to log levels for terminal output.

Vllm.LoggingUtils.NewLineFormatter

Adds logging prefix to newlines to align multi-line messages.

Vllm.LogitsProcess

Submodule bindings for vllm.logits_process.

Vllm.LogitsProcess.NoBadWordsLogitsProcessor

Wrapper for Python class NoBadWordsLogitsProcessor.

Vllm.LogitsProcess.TokenizerLike

Wrapper for Python class TokenizerLike.

Vllm.Logprobs

Submodule bindings for vllm.logprobs.

Vllm.Logprobs.FlatLogprobs

Flat logprobs of a request into multiple primitive type lists.

Vllm.Logprobs.Logprob

Infos for supporting OpenAI compatible logprobs and token ranks.

Vllm.Lora

Submodule bindings for vllm.lora.

Vllm.ModelExecutor

Submodule bindings for vllm.model_executor.

Vllm.ModelExecutor.BasevLLMParameter

Base parameter for vLLM linear layers. Extends the torch.nn.parameter

Vllm.ModelExecutor.Models.Adapters

Submodule bindings for vllm.model_executor.models.adapters.

Vllm.ModelExecutor.Models.Interfaces

Submodule bindings for vllm.model_executor.models.interfaces.

Vllm.ModelExecutor.Models.InterfacesBase

Submodule bindings for vllm.model_executor.models.interfaces_base.

Vllm.ModelExecutor.PackedvLLMParameter

Parameter for model weights which are packed on disk.

Vllm.ModelInspection

Model inspection utilities for vLLM.

Vllm.Multimodal

Submodule bindings for vllm.multimodal.

Vllm.Multimodal.Inputs

Submodule bindings for vllm.multimodal.inputs.

Vllm.Multimodal.Inputs.MultiModalFieldConfig

MultiModalFieldConfig(field: vllm.multimodal.inputs.BaseMultiModalField, modality: str)

Vllm.Multimodal.Inputs.MultiModalFieldElem

Represents a keyword argument inside a

Vllm.Multimodal.Inputs.MultiModalInputs

Represents the outputs of

Vllm.Multimodal.Inputs.MultiModalKwargsItem

A collection of

Vllm.Multimodal.Inputs.MultiModalKwargsItems

A dictionary of

Vllm.Multimodal.Inputs.PlaceholderRange

Placeholder location information for multi-modal data.

Vllm.Multimodal.MultiModalDataBuiltins

Type annotations for modality types predefined by vLLM.

Vllm.Multimodal.MultiModalHasher

Wrapper for Python class MultiModalHasher.

Vllm.Multimodal.MultiModalKwargsItems

A dictionary of

Vllm.Multimodal.MultiModalRegistry

A registry that dispatches data processing according to the model.

Vllm.Multimodal.Parse

Submodule bindings for vllm.multimodal.parse.

Vllm.Multimodal.Processing

Submodule bindings for vllm.multimodal.processing.

Vllm.Multimodal.Registry

Submodule bindings for vllm.multimodal.registry.

Vllm.Outputs

Submodule bindings for vllm.outputs.

Vllm.Outputs.ClassificationOutput

The output data of one classification output of a request.

Vllm.Outputs.ClassificationRequestOutput

The output data of a pooling request to the LLM.

Vllm.Outputs.CompletionOutput

The output data of one completion output of a request.

Vllm.Outputs.EmbeddingOutput

The output data of one embedding output of a request.

Vllm.Outputs.EmbeddingRequestOutput

The output data of a pooling request to the LLM.

Vllm.Outputs.PoolingOutput

The output data of one pooling output of a request.

Vllm.Outputs.PoolingRequestOutput

The output data of a pooling request to the LLM.

Vllm.Outputs.RequestOutput

The output data of a completion request to the LLM.

Vllm.Outputs.RequestStateStats

Stats that need to be tracked across delta updates.

Vllm.Outputs.ScoringOutput

The output data of one scoring output of a request.

Vllm.Outputs.ScoringRequestOutput

The output data of a pooling request to the LLM.

Vllm.Platforms

Submodule bindings for vllm.platforms.

Vllm.Platforms.CpuArchEnum

Enum members for CpuArchEnum.

Vllm.Platforms.Platform

Wrapper for Python class Platform.

Vllm.Platforms.PlatformEnum

Enum members for PlatformEnum.

Vllm.Plugins

Submodule bindings for vllm.plugins.

Vllm.PoolingParams

API parameters for pooling models.

Vllm.PoolingParams.Module

Submodule bindings for vllm.pooling_params.

Vllm.PoolingParams.RequestOutputKind

Enum members for RequestOutputKind.

Vllm.PoolingParamsClass

API parameters for pooling models.

Vllm.Profiler

Submodule bindings for vllm.profiler.

Vllm.Ray

Submodule bindings for vllm.ray.

Vllm.Reasoning

Submodule bindings for vllm.reasoning.

Vllm.Reasoning.ReasoningParser

Abstract reasoning parser class that should not be used directly.

Vllm.Reasoning.ReasoningParserManager

Central registry for ReasoningParser implementations.

Vllm.SamplingParams