Lorax.Config (lorax v0.2.1)
Config for Lorax.inject/2
r is the rank in the low-rank matrices used in LoRA.
A higher value of r increases the expressiveness of the adaptation,
However, it also increases the number of parameters and the computational
cost. Conversely, a lower value of r makes the adaptation simpler and less
resource-intensive. Defaults to 1.
alpha is a scaling factor that controls the magnitude of changes introduced
by the low-rank matrices. A higher value of alpha means that the
modifications made by LoRA have a greater impact on the model's original
weights. This can lead to more significant changes in the model's behavior.
A lower value results in more subtle changes. Defaults to 2.
dropout specifies the dropout rate applied to the low-rank matrices.
dropout_seed determines the seed used for Nx.Random.key/1 during
dropout application. When defined, it ensures that the LoRA adapter
produces consistent tensor values, assuming that other layers also have
deterministic outputs.
param_type specifies the numerical representation for the A and B
matrices. Defaults to float32
target_query specifies whether to apply LoRA to all query matrices in an
attention block. Defaults to true.
target_value specifies whether to apply LoRA to all value matrices in an
attention block. Defaults to true.
target_key specifies whether to apply LoRA to all key matrices in an
attention block. Defaults to true.