[go: nahoru, domu]
Skip to content

darsnack/ParameterSchedulers.jl

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

214 Commits
.github/workflows
.github/workflows
 
 
docs
docs
 
 
src
src
 
 
test
test
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
Project.toml
Project.toml
 
 
README.md
README.md
 
 

Repository files navigation

ParameterSchedulers

Dev Build Status

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

Schedule Description Type Example

Step(; start, decay, step_sizes)

Exponential decay by decay every step in step_sizes

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Step(start = 1.0, decay = 0.8, step_sizes = [2, 3, 2]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Exp(start, decay)

Exponential decay by decay every iteration

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Exp(start = 1.0, decay = 0.5) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

CosAnneal(;l0, l1, period)

Cosine annealing

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = CosAnneal(l0 = 0.0, l1 = 1.0, period = 4) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

OneCycle(nsteps, maxval)

One cycle cosine

 Complex 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = OneCycle(10, 1.0) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Triangle(l0, l1, period)

Triangle wave function

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Triangle(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

TriangleDecay2(l0, l1, period)

Triangle wave function with half the amplitude every period

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

TriangleExp(l0, l1, period, decay)

Triangle wave function with exponential amplitude decay at rate decay

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Poly(start, degree, max_iter)

Polynomial decay at degree degree.

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Poly(start = 1.0, degree = 2, max_iter = t[end]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Inv(start, decay, degree)

Inverse decay at rate (1 + t * decay)^degree

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Inv(start = 1.0, degree = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Sin(;l0, l1, period)

Sine function

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Sin(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

SinDecay2(l0, l1, period)

Sine function with half the amplitude every period

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = SinDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

SinExp(l0, l1, period)

Sine function with exponential amplitude decay at rate decay

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = SinExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

About

Common hyperparameter scheduling for ML

Resources

Readme

License

MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Julia 100.0%