View Source Bumblebee.Vision.Swin (Bumblebee v0.6.3)

Swin Transformer model.

Architectures

  • :base - plain Swin without any head on top

  • :for_image_classification - Swin tranformer model with a classification head

Global layer options

  • :output_hidden_states - when true, the model output includes all hidden states

  • :output_attentions - when true, the model output includes all attention weights

Configuration

  • :image_size - the size of the input spatial dimensions. Defaults to 224

  • :num_channels - the number of channels in the input. Defaults to 3

  • :patch_size - the size of the patch spatial dimensions. Defaults to 4

  • :embedding_size - the dimensionality of patch embedding layer. Defaults to 96

  • :use_absolute_position_embeddings - whether to add absolute position embeddings to the patch embeddings. Defaults to false

  • :num_blocks - the number of Transformer blocks in the encoder at each stage. Defaults to [2, 2, 6, 2]

  • :num_attention_heads - the number of attention heads for each attention layer in the encoder at each stage. Defaults to [3, 6, 12, 24]

  • :window_size - the window size, used to limit self-attention computation to non-overlapping windows. Defaults to 7

  • :intermediate_size_ratio - the dimensionality of the intermediate layer in the transformer feed-forward network (FFN) in the encoder, expressed as a multiplier of hidden size (at the given stage) . Defaults to 4

  • :use_attention_bias - whether to use bias in query, key, and value projections. Defaults to true

  • :activation - the activation function. Defaults to :gelu

  • :dropout_rate - the dropout rate for encoder and decoder. Defaults to 0.0

  • :attention_dropout_rate - the dropout rate for attention weights. Defaults to 0.0

  • :initializer_scale - the standard deviation of the normal initializer used for initializing kernel parameters. Defaults to 0.02

  • :drop_path_rate - the drop path rate used to for stochastic depth. Defaults to 0.1

  • :layer_norm_epsilon - the epsilon used by the layer normalization layers. Defaults to 1.0e-5

  • :num_labels - the number of labels to use in the last layer for the classification task. Defaults to 2

  • :id_to_label - a map from class index to label. Defaults to %{}

References