polaris

View Source Polaris.Schedules (polaris v0.1.0)

Parameter Schedules.

Parameter schedules are often used to anneal hyperparameters such as the learning rate during the training process. Schedules provide a mapping from the current time step to a learning rate or another hyperparameter.

Choosing a good learning rate and consequently a good learning rate schedule is typically a process of trial and error. Learning rates should be relatively small such that the learning curve does not oscillate violently during the training process, but not so small that learning proceeds too slowly. Using a schedule slowly decreases oscillations during the training process such that, as the model converges, training also becomes more stable.

All of the functions in this module are implemented as numerical functions and can be JIT or AOT compiled with any supported Nx compiler.

Link to this section Summary

Functions

constant(init_value, opts \\ [])

Constant schedule.

cosine_decay(init_value, opts \\ [])

Cosine decay schedule.

exponential_decay(init_value, opts \\ [])

Exponential decay schedule.

linear_decay(init_value, opts \\ [])

Linear decay schedule.

polynomial_decay(init_value, opts \\ [])

Polynomial schedule.

Link to this section Functions

Link to this function

constant(init_value, opts \\ [])

View Source

Constant schedule.

γ(t)=γ0\gamma(t) = \gamma_0

Link to this function

cosine_decay(init_value, opts \\ [])

View Source

Cosine decay schedule.

γ(t)=γ0∗(12(1−α)(1+cos⁡πtk)+α)\gamma(t) = \gamma_0 * \left(\frac{1}{2}(1 - \alpha)(1 + \cos\pi \frac{t}{k}) + \alpha\right)

options

 Options

  • :decay_steps - number of steps to apply decay for. kk in above formulation. Defaults to 10

  • :alpha - minimum value of multiplier adjusting learning rate. α\alpha in above formulation. Defaults to 0.0

references

 References

Link to this function

exponential_decay(init_value, opts \\ [])

View Source

Exponential decay schedule.

γ(t)=γ0∗rtk\gamma(t) = \gamma_0 * r^{\frac{t}{k}}

options

 Options

  • :decay_rate - rate of decay. rr in above formulation. Defaults to 0.95

  • :transition_steps - steps per transition. kk in above formulation. Defaults to 10

  • :transition_begin - step to begin transition. Defaults to 0

  • :staircase - discretize outputs. Defaults to false

Link to this function

linear_decay(init_value, opts \\ [])

View Source

Linear decay schedule.

options

 Options

  • :warmup - scheduler warmup steps. Defaults to 0

  • :steps - total number of decay steps. Defaults to 1000

Link to this function

polynomial_decay(init_value, opts \\ [])

View Source

Polynomial schedule.

γ(t)=(γ0−γn)∗(1−tk)p\gamma(t) = (\gamma_0 - \gamma_n) * (1 - \frac{t}{k})^p

options

 Options

  • :end_value - end value of annealed scalar. γn\gamma_n in above formulation. Defaults to 1.0e-3

  • :power - power of polynomial. pp in above formulation. Defaults to 2

  • :transition_steps - number of steps over which annealing takes place. kk in above formulation. Defaults to 10