Vllm.Logprobs (VLLM v0.3.0)

Copy Markdown View Source

Submodule bindings for vllm.logprobs.

Version

  • Requested: 0.14.0
  • Observed at generation: 0.14.0

Runtime Options

All functions accept a __runtime__ option for controlling execution behavior:

Vllm.Logprobs.some_function(args, __runtime__: [timeout: 120_000])

Supported runtime options

  • :timeout - Call timeout in milliseconds (default: 120,000ms / 2 minutes)
  • :timeout_profile - Use a named profile (:default, :ml_inference, :batch_job, :streaming)
  • :stream_timeout - Timeout for streaming operations (default: 1,800,000ms / 30 minutes)
  • :session_id - Override the session ID for this call
  • :pool_name - Target a specific Snakepit pool (multi-pool setups)
  • :affinity - Override session affinity (:hint, :strict_queue, :strict_fail_fast)

Timeout Profiles

  • :default - 2 minute timeout for regular calls
  • :ml_inference - 10 minute timeout for ML/LLM workloads
  • :batch_job - Unlimited timeout for long-running jobs
  • :streaming - 2 minute timeout, 30 minute stream_timeout

Example with timeout override

# For a long-running ML inference call
Vllm.Logprobs.predict(data, __runtime__: [timeout_profile: :ml_inference])

# Or explicit timeout
Vllm.Logprobs.predict(data, __runtime__: [timeout: 600_000])

# Route to a pool and enforce strict affinity
Vllm.Logprobs.predict(data, __runtime__: [pool_name: :strict_pool, affinity: :strict_queue])

See SnakeBridge.Defaults for global timeout configuration.

Summary

Functions

append_logprobs_for_next_position(request_logprobs, token_ids, logprobs, decoded_tokens, rank, num_logprobs, opts \\ [])

Appends logprobs for the next position

create_prompt_logprobs(flat_logprobs, opts \\ [])

Creates a container to store prompt logprobs for a request

create_sample_logprobs(flat_logprobs, opts \\ [])

Creates a container to store decode logprobs for a request

logprobs_one_position(opts \\ [])

dict() -> new empty dictionary

prompt_logprobs()

Python module attribute vllm.logprobs.PromptLogprobs.

sample_logprobs()

Python module attribute vllm.logprobs.SampleLogprobs.

Functions

append_logprobs_for_next_position(request_logprobs, token_ids, logprobs, decoded_tokens, rank, num_logprobs, opts \\ [])

@spec append_logprobs_for_next_position(
  term(),
  [integer()],
  [float()],
  term(),
  integer(),
  integer(),
  keyword()
) :: {:ok, nil} | {:error, Snakepit.Error.t()}

Appends logprobs for the next position

Parameters

  • request_logprobs (term())
  • token_ids (list(integer()))
  • logprobs (list(float()))
  • decoded_tokens (term())
  • rank (integer())
  • num_logprobs (integer())

Returns

  • nil

create_prompt_logprobs(flat_logprobs, opts \\ [])

@spec create_prompt_logprobs(
  boolean(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

Creates a container to store prompt logprobs for a request

Parameters

  • flat_logprobs (boolean())

Returns

  • term()

create_sample_logprobs(flat_logprobs, opts \\ [])

@spec create_sample_logprobs(
  boolean(),
  keyword()
) :: {:ok, term()} | {:error, Snakepit.Error.t()}

Creates a container to store decode logprobs for a request

Parameters

  • flat_logprobs (boolean())

Returns

  • term()

logprobs_one_position(opts \\ [])

@spec logprobs_one_position(keyword()) :: {:ok, term()} | {:error, Snakepit.Error.t()}

dict() -> new empty dictionary

dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable:

  d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

Parameters

  • args (term())
  • kwargs (term())

Returns

  • term()

prompt_logprobs()

@spec prompt_logprobs() :: {:ok, term()} | {:error, Snakepit.Error.t()}

Python module attribute vllm.logprobs.PromptLogprobs.

Returns

  • term()

sample_logprobs()

@spec sample_logprobs() :: {:ok, term()} | {:error, Snakepit.Error.t()}

Python module attribute vllm.logprobs.SampleLogprobs.

Returns

  • term()