OpenVLA: Open-Source Vision-Language-Action Model.

OpenVLA is a vision-language-action model for robot manipulation that combines a DINOv2 vision encoder with a Llama-style language model backbone to predict discretized robot actions from images and text instructions.

Key Innovation: Action Tokenization

Instead of predicting continuous actions directly, OpenVLA discretizes each action dimension into bins (default 256), treating robot control as a sequence-to-sequence problem. The model autoregressively generates action tokens conditioned on visual and language tokens.

Architecture

Image [batch, C, H, W]           Text "pick up the red block"
      |                                    |
      v                                    v
+================+                  +================+
|   DINOv2 ViT   |                  |  LLM Tokenizer |
+================+                  +================+
      |                                    |
      v                                    v
Visual Tokens                       Text Token IDs
[batch, num_patches, vision_dim]   [batch, text_len]
      |                                    |
      v                                    v
[Vision-LM Projection]             [LLM Embedding]
[batch, num_patches, hidden_dim]   [batch, text_len, hidden_dim]
      |                                    |
      +---------------+--------------------+
                      |
                      v
            [Concatenate: visual | text]
            [batch, num_patches + text_len, hidden_dim]
                      |
                      v
            +====================+
            |  Llama-style LLM   |
            |  (Decoder-Only)    |
            |  with Causal Mask  |
            +====================+
                      |
                      v
            Action Token Logits
            [batch, action_dim, num_bins]
                      |
                      v
            Argmax -> Detokenize
                      |
                      v
            Continuous Actions
            [batch, action_dim]

Action Tokenization

For action dimension d with range [a_min, a_max]:

• Bin index = floor((action - a_min) / (a_max - a_min) * (num_bins - 1))
• Action = a_min + (bin_index / (num_bins - 1)) * (a_max - a_min)

Default: 256 bins per dimension, covering normalized range [-1, 1].

Usage

model = OpenVLA.build(
  image_size: 224,
  vision_encoder: :dino_v2,
  action_dim: 7,
  num_action_bins: 256,
  hidden_dim: 2048,
  num_layers: 24,
  num_heads: 16
)

# Forward pass: image + text -> action logits
action_logits = model.predict(params, %{"image" => img, "text_tokens" => tokens})

# Tokenize ground truth actions for training
action_tokens = OpenVLA.tokenize_actions(actions, 256)

# Compute loss (cross-entropy on action tokens only)
loss = OpenVLA.vla_loss(action_logits, action_tokens)

# Detokenize predictions for inference
actions = OpenVLA.detokenize_actions(predicted_tokens, 256)

References

• Paper: "OpenVLA: An Open-Source Vision-Language-Action Model" (Kim et al., 2024) - https://arxiv.org/abs/2406.09246
• Project: https://openvla.github.io/