Vllm.Sequence.IntermediateTensors (VLLM v0.3.0)

Copy Markdown View Source

For all pipeline stages except the last, we need to return the hidden

states and residuals to be sent to the next stage. This data structure contains the hidden states and residuals for a request.

Each stage also needs to handle its own kv_connector_output.

Summary

Types

t()

Functions

items(ref, opts \\ [])

Sequence and its related classes.

new(tensors, args, opts \\ [])

Initialize self. See help(type(self)) for accurate signature.

Types

t()

@opaque t()

Functions

items(ref, opts \\ [])

@spec items(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

Sequence and its related classes.

Returns

  • term()

new(tensors, args, opts \\ [])

@spec new(%{optional(String.t()) => term()}, [term()], keyword()) ::
  {:ok, SnakeBridge.Ref.t()} | {:error, Snakepit.Error.t()}

Initialize self. See help(type(self)) for accurate signature.

Parameters

  • tensors (%{optional(String.t()) => term()})
  • kv_connector_output (term() default: None)