SigLIP - Sigmoid Loss for Language-Image Pre-training.

Implements SigLIP, which replaces the softmax-based contrastive loss (used in CLIP) with a simpler sigmoid-based binary classification loss. Each image-text pair is treated as an independent binary classification problem.

Key Innovation

Instead of softmax cross-entropy over all pairs:

CLIP: -log(exp(sim_pos) / sum(exp(sim_all)))

SigLIP uses sigmoid for each pair independently:

SigLIP: sum_ij(log(sigmoid(t * sim_ij * y_ij)))

Where:

• sim_ij is the cosine similarity between embeddings i and j
• y_ij = +1 for matching pairs, -1 for non-matching pairs
• t is a learnable temperature parameter

Advantages

• Simpler: No need to normalize over all negatives
• Scalable: Each pair is independent, enabling larger batch sizes
• Stable: Sigmoid gradients are bounded, unlike softmax with temperature
• Effective: Matches or exceeds CLIP performance in practice

Architecture

Image Encoder          Text Encoder
      |                      |
      v                      v
+------------+        +------------+
|  Backbone  |        |  Backbone  |
+------------+        +------------+
      |                      |
      v                      v
+------------+        +------------+
| Projection |        | Projection |
+------------+        +------------+
      |                      |
      v                      v
     z_img    SigLIP       z_txt
      |       Loss           |
      +-------> <-----------+

Usage

{encoder, _} = SigLIP.build(input_dim: 512, projection_dim: 256)

# Compute SigLIP loss
loss = SigLIP.loss(z_img, z_txt, temperature: 1.0)

Reference

• "Sigmoid Loss for Language Image Pre-Training" (Zhai et al., 2023)