Vllm.Config.KVTransferConfig (VLLM v0.3.0)

Copy Markdown View Source

Configuration for distributed KV cache transfer.

Summary

Types

t()

Functions

compute_hash(ref, opts \\ [])

WARNING: Whenever a new field is added to this config,

enable_permute_local_kv(ref)
engine_id(ref)
get_from_extra_config(ref, key, default, opts \\ [])

Python method KVTransferConfig.get_from_extra_config.

is_kv_consumer(ref)
is_kv_producer(ref)
is_kv_transfer_instance(ref)
kv_buffer_device(ref)
kv_buffer_size(ref)
kv_connector(ref)
kv_connector_module_path(ref)
kv_ip(ref)
kv_load_failure_policy(ref)
kv_parallel_size(ref)
kv_port(ref)
kv_rank(ref)
kv_role(ref)
new(dataclass_self__, args, kwargs, opts \\ [])

Constructs KVTransferConfig.

Types

t()

@opaque t()

Functions

compute_hash(ref, opts \\ [])

@spec compute_hash(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, String.t()} | {:error, Snakepit.Error.t()}

WARNING: Whenever a new field is added to this config,

ensure that it is included in the factors list if it affects the computation graph.

Provide a hash that uniquely identifies all the configs that affect the structure of the computation graph from input ids/embeddings to the final hidden states, excluding anything before input ids/embeddings and after the final hidden states.

Returns

  • String.t()

enable_permute_local_kv(ref)

@spec enable_permute_local_kv(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

engine_id(ref)

@spec engine_id(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

get_from_extra_config(ref, key, default, opts \\ [])

@spec get_from_extra_config(SnakeBridge.Ref.t(), term(), term(), keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

Python method KVTransferConfig.get_from_extra_config.

Parameters

  • key (term())
  • default (term())

Returns

  • term()

is_kv_consumer(ref)

@spec is_kv_consumer(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

is_kv_producer(ref)

@spec is_kv_producer(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

is_kv_transfer_instance(ref)

@spec is_kv_transfer_instance(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_buffer_device(ref)

@spec kv_buffer_device(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_buffer_size(ref)

@spec kv_buffer_size(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_connector(ref)

@spec kv_connector(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_connector_module_path(ref)

@spec kv_connector_module_path(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_ip(ref)

@spec kv_ip(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

kv_load_failure_policy(ref)

@spec kv_load_failure_policy(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_parallel_size(ref)

@spec kv_parallel_size(SnakeBridge.Ref.t()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

kv_port(ref)

@spec kv_port(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

kv_rank(ref)

@spec kv_rank(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

kv_role(ref)

@spec kv_role(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

new(dataclass_self__, args, kwargs, opts \\ [])

@spec new(term(), term(), term(), keyword()) ::
  {:ok, SnakeBridge.Ref.t()} | {:error, Snakepit.Error.t()}

Constructs KVTransferConfig.

Parameters

  • dataclass_self__ (term())
  • args (term())
  • kwargs (term())