Vllm.Logprobs.FlatLogprobs (VLLM v0.3.0)

Copy Markdown View Source

Flat logprobs of a request into multiple primitive type lists.

Compared to list[dict[int, Logprob]], this data structure reduced GC overhead significantly. As it flattened logprob information for all positions and ranks in to multiple primitive type lists (i.e. logprobs, token_ids, ranks per token_ids, decoded_tokens). So regardless of the sequence length and top_logprobs setup, FlatLogprobs would only introduce a constant amount of objects.

As each position might contains different amount of ranks, start_indices_per_position would be used to access the logprob ranges for different positions.

NOTE: To reduce the migration overhead and improve backward compatibility, we support the key Sequence APIs of list, so it could act as list[LogprobsOnePosition]

Summary

Types

t()

Functions

append(ref, logprobs_one_position, opts \\ [])

Appends the container with logprobs for the next position

append_fast(ref, token_ids, logprobs, ranks, decoded_tokens, opts \\ [])

Appends logprobs for the next position without creating

clear(ref, opts \\ [])

S.clear() -> None -- remove all items from S

count(ref, value, opts \\ [])

S.count(value) -> integer -- return number of occurrences of value

extend(ref, logprobs_multi_positions, opts \\ [])

Extends the container with logprobs for the next multiple positions

index(ref, value, args, opts \\ [])

S.index(value, [start, [stop]]) -> integer -- return first index of value.

insert(ref, item, opts \\ [])

S.insert(index, value) -- insert value before index

new(args, opts \\ [])

Initialize self. See help(type(self)) for accurate signature.

pop(ref, args, opts \\ [])

S.pop([index]) -> item -- remove and return item at index (default last).

remove(ref, value, opts \\ [])

S.remove(value) -- remove first occurrence of value.

reverse(ref, opts \\ [])

S.reverse() -- reverse IN PLACE

Types

t()

@opaque t()

Functions

append(ref, logprobs_one_position, opts \\ [])

@spec append(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, nil} | {:error, Snakepit.Error.t()}

Appends the container with logprobs for the next position

Parameters

  • logprobs_one_position (term())

Returns

  • nil

append_fast(ref, token_ids, logprobs, ranks, decoded_tokens, opts \\ [])

@spec append_fast(
  SnakeBridge.Ref.t(),
  [integer()],
  [float()],
  term(),
  term(),
  keyword()
) ::
  {:ok, nil} | {:error, Snakepit.Error.t()}

Appends logprobs for the next position without creating

the intermediate logprob dictionary.

Parameters

  • token_ids (list(integer()))
  • logprobs (list(float()))
  • ranks (term())
  • decoded_tokens (term())

Returns

  • nil

clear(ref, opts \\ [])

@spec clear(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

S.clear() -> None -- remove all items from S

Returns

  • term()

count(ref, value, opts \\ [])

@spec count(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

S.count(value) -> integer -- return number of occurrences of value

Parameters

  • value (term())

Returns

  • term()

extend(ref, logprobs_multi_positions, opts \\ [])

@spec extend(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, nil} | {:error, Snakepit.Error.t()}

Extends the container with logprobs for the next multiple positions

Parameters

  • logprobs_multi_positions (term())

Returns

  • nil

index(ref, value, args, opts \\ [])

@spec index(SnakeBridge.Ref.t(), term(), [term()], keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

S.index(value, [start, [stop]]) -> integer -- return first index of value.

Raises ValueError if the value is not present.

Supporting start and stop arguments is optional, but recommended.

Parameters

  • value (term())
  • start (term() default: 0)
  • stop (term() default: None)

Returns

  • term()

insert(ref, item, opts \\ [])

@spec insert(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, nil} | {:error, Snakepit.Error.t()}

S.insert(index, value) -- insert value before index

Parameters

  • item (term())

Returns

  • nil

new(args, opts \\ [])

@spec new(
  [term()],
  keyword()
) :: {:ok, SnakeBridge.Ref.t()} | {:error, Snakepit.Error.t()}

Initialize self. See help(type(self)) for accurate signature.

Parameters

  • start_indices (list(integer()) default: <factory>)
  • end_indices (list(integer()) default: <factory>)
  • token_ids (list(integer()) default: <factory>)
  • logprobs (list(float()) default: <factory>)
  • ranks (list(term()) default: <factory>)
  • decoded_tokens (list(term()) default: <factory>)

pop(ref, args, opts \\ [])

@spec pop(SnakeBridge.Ref.t(), [term()], keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

S.pop([index]) -> item -- remove and return item at index (default last).

Raise IndexError if list is empty or index is out of range.

Parameters

  • index (term() default: -1)

Returns

  • term()

remove(ref, value, opts \\ [])

@spec remove(SnakeBridge.Ref.t(), term(), keyword()) ::
  {:ok, term()} | {:error, Snakepit.Error.t()}

S.remove(value) -- remove first occurrence of value.

Raise ValueError if the value is not present.

Parameters

  • value (term())

Returns

  • term()

reverse(ref, opts \\ [])

@spec reverse(
  SnakeBridge.Ref.t(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

S.reverse() -- reverse IN PLACE

Returns

  • term()