View Source Axon.Schedules (Axon v0.3.1)
Parameter Schedules.
Parameter schedules are often used to anneal hyperparameters such as the learning rate during the training process. Schedules provide a mapping from the current time step to a learning rate or another hyperparameter.
Choosing a good learning rate and consequently a good learning rate schedule is typically a process of trial and error. Learning rates should be relatively small such that the learning curve does not oscillate violently during the training process, but not so small that learning proceeds too slowly. Using a schedule slowly decreases oscillations during the training process such that, as the model converges, training also becomes more stable.
All of the functions in this module are implemented as
numerical functions and can be JIT or AOT compiled with
any supported Nx compiler.
Link to this section Summary
Functions
Constant schedule.
Cosine decay schedule.
Exponential decay schedule.
Polynomial schedule.
Link to this section Functions
Constant schedule.
$$\gamma(t) = \gamma_0$$
options
Options
:init_value- initial value. $\gamma_0$ in above formulation. Defaults to1.0e-2
Cosine decay schedule.
$$\gamma(t) = \gamma_0 (1 - \alpha)(\frac{1}{2}(1 + \cos{\pi \frac{t}{k}})) + \alpha$$
options
Options
:init_value- initial value. $\gamma_0$ in above formulation. Defaults to1.0e-2:decay_steps- number of steps to apply decay for. $k$ in above formulation. Defaults to10:alpha- minimum value of multiplier adjusting learning rate. $\alpha$ in above formulation. Defaults to0.0
references
References
Exponential decay schedule.
$$\gamma(t) = \gamma_0 * r^{\frac{t}{k}}$$
options
Options
:init_value- initial value. $\gamma$ in above formulation. Defaults to1.0e-2:decay_rate- rate of decay. $r$ in above formulation. Defaults to0.95:transition_steps- steps per transition. $k$ in above formulation. Defaults to10:transition_begin- step to begin transition. Defaults to0:staircase- discretize outputs. Defaults tofalse
Polynomial schedule.
$$\gamma(t) = (\gamma_0 - \gamma_n) * (1 - \frac{t}{k})^p$$
options
Options
:init_value- initial value. $\gamma_0$ in above formulation. Defaults to1.0e-2:end_value- end value of annealed scalar. $\gamma_n$ in above formulation. Defaults to1.0e-3:power- power of polynomial. $p$ in above formulation. Defaults to2:transition_steps- number of steps over which annealing takes place. $k$ in above formulation. Defaults to10