Batch descriptor for cudagraph dispatching. We should keep the num of
items as minimal as possible to properly and uniquely describe the padded batch for cudagraph.
Summary
Functions
Return a new dict which maps field names to their values.
Make a new BatchDescriptor object from a sequence or iterable
Return a new BatchDescriptor object replacing specified fields with new values
Return number of occurrences of value.
Return first index of value.
Initialize self. See help(type(self)) for accurate signature.
Return a relaxed version of current batch descriptor that is still compatible
Types
Functions
@spec _asdict( SnakeBridge.Ref.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Return a new dict which maps field names to their values.
Returns
term()
@spec _make(SnakeBridge.Ref.t(), term(), keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Make a new BatchDescriptor object from a sequence or iterable
Parameters
iterable(term())
Returns
term()
@spec _replace( SnakeBridge.Ref.t(), keyword() ) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Return a new BatchDescriptor object replacing specified fields with new values
Parameters
kwds(term())
Returns
term()
@spec count(SnakeBridge.Ref.t(), term(), keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Return number of occurrences of value.
Parameters
value(term())
Returns
term()
@spec has_lora(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec index(SnakeBridge.Ref.t(), term(), [term()], keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
Return first index of value.
Raises ValueError if the value is not present.
Parameters
value(term())start(term() default: 0)stop(term() default: 9223372036854775807)
Returns
term()
@spec new( [term()], keyword() ) :: {:ok, SnakeBridge.Ref.t()} | {:error, Snakepit.Error.t()}
Initialize self. See help(type(self)) for accurate signature.
Parameters
args(term())kwargs(term())
@spec num_reqs(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec num_tokens(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}
@spec relax_for_mixed_batch_cudagraphs( SnakeBridge.Ref.t(), keyword() ) :: {:ok, t()} | {:error, Snakepit.Error.t()}
Return a relaxed version of current batch descriptor that is still compatible
with PIECEWISE cudagraphs (or mixed prefill-decode FA cudagraphs).
Returns
Vllm.ForwardContext.BatchDescriptor.t()
@spec uniform(SnakeBridge.Ref.t()) :: {:ok, term()} | {:error, Snakepit.Error.t()}