Vllm.ForwardContext.BatchDescriptor (VLLM v0.3.0)

Copy Markdown View Source

Batch descriptor for cudagraph dispatching. We should keep the num of

items as minimal as possible to properly and uniquely describe the padded batch for cudagraph.

Summary

Functions

Return a new dict which maps field names to their values.

Make a new BatchDescriptor object from a sequence or iterable

Return a new BatchDescriptor object replacing specified fields with new values

Return number of occurrences of value.

Return first index of value.

Initialize self. See help(type(self)) for accurate signature.

Return a relaxed version of current batch descriptor that is still compatible

Types

t()

@opaque t()

Functions

_asdict(ref, opts \\ [])

@spec _asdict(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

Return a new dict which maps field names to their values.

Returns

  • term()

_make(ref, iterable, opts \\ [])

@spec _make(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

Make a new BatchDescriptor object from a sequence or iterable

Parameters

  • iterable (term())

Returns

  • term()

_replace(ref, opts \\ [])

@spec _replace(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

Return a new BatchDescriptor object replacing specified fields with new values

Parameters

  • kwds (term())

Returns

  • term()

count(ref, value, opts \\ [])

@spec count(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

Return number of occurrences of value.

Parameters

  • value (term())

Returns

  • term()

has_lora(ref)

@spec has_lora(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

index(ref, value, args, opts \\ [])

@spec index(SnakeBridge.Ref.t(), term(), [term()], keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

Return first index of value.

Raises ValueError if the value is not present.

Parameters

  • value (term())
  • start (term() default: 0)
  • stop (term() default: 9223372036854775807)

Returns

  • term()

new(args, opts \\ [])

@spec new(
  [term()],
  keyword()
) :: {:ok, SnakeBridge.Ref.t()} | {:error, Snakepit.Error.t()}

Initialize self. See help(type(self)) for accurate signature.

Parameters

  • args (term())
  • kwargs (term())

num_reqs(ref)

@spec num_reqs(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

num_tokens(ref)

@spec num_tokens(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

relax_for_mixed_batch_cudagraphs(ref, opts \\ [])

@spec relax_for_mixed_batch_cudagraphs(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, t()} | {:error, Snakepit.Error.t()}

Return a relaxed version of current batch descriptor that is still compatible

with PIECEWISE cudagraphs (or mixed prefill-decode FA cudagraphs).

Returns

  • Vllm.ForwardContext.BatchDescriptor.t()

uniform(ref)

@spec uniform(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}