learning rates neural nets

Keras learning rate schedules and decay

In this tutorial, you will study learning rate schedules and decay using Keras. You’ll discover ways to use Keras’ commonplace learning rate decay along with step-based, linear, and polynomial learning rate schedules.

When coaching a neural community, the learning rate is usually crucial hyperparameter for you to tune:

  • Too small a learning rate and your neural network might not study in any respect
  • Too giant a learning rate and chances are you’ll overshoot areas of low loss (and even overfit from the start of coaching)

On the subject of coaching a neural network, probably the most bang in your buck (when it comes to accuracy) is going to return from choosing the right learning rate and applicable learning rate schedule.

But that’s simpler stated than completed.

To help deep learning practitioners reminiscent of yourself discover ways to assess a problem and select an applicable learning rate, we’ll be beginning a collection of tutorials on learning rate schedules, decay, and hyperparameter tuning with Keras.

By the top of this collection, you’ll have a superb understanding of tips on how to appropriately and successfully apply learning rate schedules with Keras to your personal deep learning tasks.

To discover ways to use Keras for learning rate schedules and decay, just hold studying

Keras learning rate schedules and decay

Within the first a part of this information, we’ll talk about why the learning rate is an important hyperparameter on the subject of coaching your personal deep neural networks.

We’ll then dive into why we might need to modify our learning rate throughout training.

From there I’ll present you learn how to implement and utilize numerous learning rate schedules with Keras, together with:

  • The decay schedule constructed into most Keras optimizers
  • Step-based learning rate schedules
  • Linear learning rate decay
  • Polynomial learning rate schedules

We’ll then carry out quite a few experiments on the CIFAR-10 utilizing these learning rate schedules and evaluate which one performed the perfect.

These sets of experiments will serve as a template you should use when exploring your personal deep learning tasks and choosing an applicable learning rate and learning rate schedule.

Why modify our learning rate and use learning rate schedules?

To see why learning rate schedules are a worthwhile technique to use to help improve model accuracy and descend into areas of decrease loss, contemplate the standard weight update formulation used by almost all neural networks:

W += alpha * gradient

Recall that the learning rate, alpha, controls the “step” we make alongside the gradient. Bigger values of alpha suggest that we’re taking greater steps. Whereas smaller values of alpha will make tiny steps. If alpha is zero the community can’t make any steps in any respect (because the gradient multiplied by zero is zero).

Most preliminary learning charges (however not all) you encounter are sometimes within the set alpha = 1e^-1, 1e^-2, 1e^-3 .

A network is then educated for a hard and fast variety of epochs without altering the learning rate.

This technique may fit nicely in some situations, nevertheless it’s typically useful to lower our learning rate over time. When coaching our network, we’re trying to find some location alongside our loss panorama the place the network obtains affordable accuracy. It doesn’t should be a worldwide minima or perhaps a native minima, however in apply, merely discovering an area of the loss panorama with fairly low loss is “good enough”.

If we continuously hold a learning rate excessive, we might overshoot these areas of low loss as we’ll be taking too giant of steps to descend into those collection.

As an alternative, what we will do is decrease our learning rate, thereby permitting our network to take smaller steps — this decreased learning rate allows our network to descend into areas of the loss panorama which are “more optimal” and would have otherwise been missed completely by our learning rate learning.

We will, subsequently, view the process of learning rate scheduling as:

  1. Finding a set of fairly “good” weights early within the training process with a bigger learning rate.
  2. Tuning these weights later in the process to seek out extra optimal weights using a smaller learning rate.

We’ll be masking a number of the most popular learning rate schedules in this tutorial.

Venture structure

When you’ve grabbed and extracted the “Downloads” go forward and use the
tree  command to examine the venture folder:

output/  listing will include learning rate and training history plots. The five experiments included in the results section correspond to the five plots with the
train_*.png  filenames, respectively.

pyimagesearch  module incorporates our ResNet CNN and our
learning_rate_schedulers.py . The
LearningRateDecay  dad or mum class merely includes a technique referred to as
plot  for plotting each of our kinds of learning rate decay. Additionally included are subclasses,
StepDecay  and
PolynomialDecay  which calculate the learning rate upon the completion of every epoch. Both of these courses include the
plot  technique by way of inheritance (an object-oriented idea).

Our training script,
practice.py , will practice ResNet on the CIFAR-10 dataset. We’ll run the script with the absence of learning rate decay as well as commonplace, linear, step-based, and polynomial learning rate decay.

The usual “decay” schedule in Keras

The Keras library ships with a time-based learning rate scheduler — it is controlled by way of the
decay  parameter of the optimizer class (comparable to
Adam, and so forth.).

To discover how we will make the most of this sort of learning rate decay, let’s check out an instance of how we might initialize the ResNet architecture and the SGD optimizer:

Here we initialize our SGD optimizer with an preliminary learning rate of
1e-2 . We then set our
decay  to be the learning rate divided by the whole number of epochs we’re coaching the network for (a standard rule of thumb).

Internally, Keras applies the next learning rate schedule to adjust the learning rate after every batch replace — it is a false impression that Keras updates the standard decay after every epoch. Maintain this in thoughts when utilizing the default learning rate scheduler provided with Keras.

The replace components follows: lr = init_lr * frac1.01.0 + decay * iterations

Utilizing the CIFAR-10 dataset for instance, we’ve got a complete of 50,000 training photographs.

If we use a batch measurement of
64 , that suggests there are a complete of lceil50000 / 64rceil = 782 steps per epoch. Subsequently, a complete of
782  weight updates have to be applied before an epoch completes.

To see an example of the learning rate schedule calculation, let’s assume our initial learning rate is alpha = 0.01 and our decay = frac0.0140 (with the idea that we are training for forty epochs).

The learning rate at step zero, before any learning rate schedule has been applied, is:

lr = 0.01 * frac1.01.0 + 0.00025 * (0 * 782) = 0.01

Firstly of epoch one we will see the following learning rate:

lr = 0.01 * frac1.0(1.0 + 0.00025 * (1 * 782) = 0.00836

Figure 1 under continues the calculation of Keras’ normal learning rate decay alpha =0.01 and a decay of frac0.0140:

Figure 1: Keras’ normal learning rate decay desk.

You’ll discover ways to utilize any such learning rate decay inside the “Implementing our training script” and “Keras learning rate schedule results” sections of this publish, respectively.

Our LearningRateDecay class

In the the rest of this tutorial, we’ll be implementing our personal customized learning rate schedules and then incorporating them with Keras when coaching our neural networks.

To keep our code neat and tidy, and to not mention, comply with object-oriented programming greatest practices, let’s first outline a base
LearningRateDecay  class that we’ll subclass for every respective learning rate schedule.

Open up the
learning_rate_schedulers.py  in your directory structure and insert the following code:

Every and each learning rate schedule we implement could have a plot perform, enabling us to visualise our learning rate over time.

With our base
LearningRateSchedule  class implement, let’s move on to making a step-based learning rate schedule.

Step-based learning rate schedules with Keras

Figure 2: Keras learning rate step-based decay. The schedule in pink is a decay issue of 0.5 and blue is a factor of 0.25.

One widespread learning rate scheduler is step-based decay where we systematically drop the learning rate after specific epochs throughout coaching.

The step decay learning rate scheduler may be seen as a piecewise perform, as visualized in Determine 2 — right here the learning rate is fixed for a lot of epochs, then drops, is constant as soon as extra, then drops again, and so forth.

When making use of step decay to our learning rate, we have now two choices:

  1. Define an equation that models the piecewise drop-in learning rate that we wish to obtain.
  2. Use what I name the
    ctrl + c technique to train a deep neural community. Right here we practice for some number of epochs at a given learning rate and ultimately discover validation performance stagnating/stalling, then 
    ctrl + c to cease the script, regulate our learning rate, and continue training.

We’ll primarily be focusing on the equation-based piecewise drop to learning rate scheduling in this publish.

ctrl + c technique is a bit more advanced and usually applied to larger datasets using deeper neural networks the place the exact number of epochs required to acquire an inexpensive model is unknown.

When you’d wish to study more concerning the
ctrl + c technique to training, please seek advice from Deep Learning for Pc Imaginative and prescient with Python.

When applying step decay, we frequently drop our learning rate by both (1) half or (2) an order of magnitude after every fastened variety of epochs. For instance, let’s suppose our preliminary learning rate is alpha = 0.01.

After 10 epochs we drop the learning rate to alpha = 0.005.

After one other 10 epochs (i.e., the 20th complete epoch), alpha is dropped by an element of
zero.5  again, such that alpha = 0.0025, and so on.

The truth is, this is the very same learning rate schedule that is depicted in Determine 2 (purple line).

The blue line shows a extra aggressive drop issue of
zero.25 . Modeled mathematically, we will outline our step-based decay equation as:

alpha_E + 1 = alpha_I times F^(1 + E) / D

The place alpha_I is the initial learning rate, F is the factor worth controlling the rate by which the learning date drops, D is the “Drop every” epochs worth, and E is the current epoch.

The larger our issue F is, the slower the learning rate will decay.

Conversely, the smaller the factor F, the quicker the learning rate will decay.

All that stated, let’s go forward and implement our
StepDecay  class now.

Go back to your
learning_rate_schedulers.py  file and insert the following code:

Line 20 defines the constructor to our
StepDecay  class. We then store the initial learning rate (
initAlpha ), drop factor, and
dropEvery  epochs values (Strains 23-25).

__call__ perform:

  • Accepts the current
    epoch  number.
  • Computes the learning rate based mostly on the step-based decay formulation detailed above (Strains 29 and 30).
  • Returns the computed learning rate for the current epoch (Line 33).

You’ll see easy methods to use this learning rate schedule later on this publish.

Linear and polynomial learning rate schedules in Keras

Two of my favorite learning rate schedules are linear learning rate decay and polynomial learning rate decay.

Utilizing these methods our learning rate is decayed to zero over a hard and fast number of epochs.

The rate through which the learning rate is decayed is predicated on the parameters to the polynomial perform. A smaller exponent/energy to the polynomial will cause the learning rate to decay “more slowly”, whereas larger exponents decay the learning rate “more quickly”.

Conveniently, both of these strategies could be carried out in a single class:

Line 36 defines the constructor to our
PolynomialDecay  class which requires three values:

  • maxEpochs : The entire number of epochs we’ll be coaching for.

  • initAlpha : The preliminary learning rate.

  • power : The facility/exponent of the polynomial.

Notice that in the event you set
power=1.zero  then you will have a linear learning rate decay.

Strains 45 and 46 compute the adjusted learning rate for the current epoch while Line 49 returns the new learning rate.

Implementing our coaching script

Now that we’ve carried out a couple of totally different Keras learning rate schedules, let’s see how we will use them inside an precise training script.

Create a file named 
practice.py  file in your editor and insert the following code:

Strains 2-16 import required packages. Line three units the
matplotlib  backend in order that we will create plots as picture information. Our most notable imports embrace:

  • StepDecay : Our class which calculates and plots step-based learning rate decay.

  • PolynomialDecay : The category we wrote to calculate polynomial-based learning rate decay.

  • ResNet : Our Convolutional Neural Community carried out in Keras.

  • LearningRateScheduler : A Keras callback. We’ll move our learning rate
    schedule  to this class which will probably be referred to as as a callback at the completion of every epoch to calculate our learning rate.

Let’s transfer on and parse our command line arguments:

Our script accepts any of 4 command line arguments when the script known as by way of the terminal:

  • –schedule : The learning rate schedule technique. Legitimate choices are “standard”, “step”, “linear”, “poly”. By default, no learning rate schedule shall be used.

  • –epochs : The variety of epochs to train for (
    default=100 ).

  • –lr-plot : The path to the output plot. I recommend overriding the
    default  of
    lr.png  with a more descriptive path + filename.

  • –train-plot : The trail to the output accuracy/loss coaching historical past plot. Once more, I recommend a descriptive path + filename, otherwise
    training.png  shall be set by
    default .

With our imports and command line arguments in hand, now it’s time to initialize our learning rate schedule:

Line 33 sets the variety of
epochs  we’ll practice for instantly from the command line
args  variable. From there we’ll initialize our
callbacks  listing and learning rate
schedule  (Strains 34 and 35).

Strains 38-50 then choose the learning rate
schedule  if
args[“schedule”] incorporates a legitimate value:

  • “step” : Initializes
    StepDecay .

  • “linear” : Initializes
    PolynomialDecay  with
    power=1  indicating that a linear learning rate decay might be utilized.

  • “poly” : 
    PolynomialDecay  with a
    power=5  will probably be used.

After you’ve reproduced the outcomes of the experiments on this tutorial, make sure you revisit Strains 38-50 and insert further
elif  statements of your personal so you possibly can run some of your personal experiments!

Strains 54 and 55 initialize the
LearningRateScheduler  with the schedule as a single callback part of the
callbacks  listing. There’s a case the place no learning rate decay will probably be used (i.e. if the
–schedule  command line argument isn’t overridden when the script is executed).

Let’s go ahead and load our knowledge:

Line 60 masses our CIFAR-10 knowledge. The dataset is conveniently already cut up into coaching and testing sets.

The one preprocessing we must perform is to scale the info into the vary [0, 1] (Strains 61 and 62).

Strains 65-67 binarize the labels and then Strains 70 and 71 initialize our
labelNames  (i.e. courses). Do not add to or alter the
labelNames  record as order and size of the record matter.

Let’s initialize
decay parameter:

Line 74 initializes our learning rate
decay .

If we’re utilizing the
“standard”  learning rate decay schedule, then the decay is initialized as
1e-1 / epochs  (Strains 78-80).

With all of our initializations taken care of, let’s go ahead and compile + practice our
ResNet  model:

Our Stochastic Gradient Descent (
SGD ) optimizer is initialized on Line 87 utilizing our
decay .

From there, Strains 88 and 89 build our
ResNet  CNN with an input form of 32x32x3 and 10 courses. For an in-depth evaluation of ResNet, ensure confer with Chapter 10: ResNet of Deep Learning for Pc Vision with Python.

mannequin  is compiled with a
loss  perform of
“categorical_crossentropy”  since our dataset has > 2 courses. In case you use a unique dataset with only 2 courses, remember to use
loss=”binary_crossentropy” .

Strains 94 and 95 kick of our coaching course of. Discover that we’ve offered the
callbacks  as a parameter. The
callbacks  will probably be referred to as when each epoch is accomplished. Our
LearningRateScheduler  contained therein will deal with our learning rate decay (as long as
callbacks  isn’t an empty listing).

Finally, let’s consider our network and generate plots:

Strains 99-101 evaluate our community and print a classification report back to our terminal.

Strains 104-115 generate and save our training historical past plot (accuracy/loss curves). Strains 119-121 generate a learning rate schedule plot, if applicable. We’ll examine these plot visualizations in the next part.

Keras learning rate schedule outcomes

With each our (1) learning rate schedules and (2) training scripts carried out, let’s run some experiments to see which learning rate schedule will carry out greatest given:

  1. An preliminary learning rate of
  2. Coaching for a complete of
    100  epochs

Experiment #1: No learning rate decay/schedule

As a baseline, let’s first practice our ResNet mannequin on CIFAR-10 with no learning rate decay or schedule:

Figure 3: Our first experiment for coaching ResNet on CIFAR-10 does not have learning rate decay.

Here we acquire ~85% accuracy, however as we will see, validation loss and accuracy stagnate previous epoch ~15 and do not enhance over the remainder of the 100 epochs.

Our aim is now to make the most of learning rate scheduling to beat our 85% accuracy (with out overfitting).

Experiment: #2: Keras normal optimizer learning rate decay

In our second experiment we are going to use Keras’ commonplace decay-based learning rate schedule:

Determine four: Our second learning rate decay schedule experiment makes use of Keras’ commonplace learning rate decay schedule.

This time we only get hold of 82% accuracy, which matches to point out, learning rate decay/scheduling won’t all the time enhance your results! It is advisable be careful which learning rate schedule you make the most of.

Experiment #three: Step-based learning rate schedule results

Let’s go ahead and carry out step-based learning rate scheduling which can drop our learning rate by an element of zero.25 every 15 epochs:

Figure 5: Experiment #3 demonstrates a step-based learning rate schedule (left). The training historical past accuracy/loss curves are proven on the suitable.

Figure 5 (left) visualizes our learning rate schedule. Discover how after each 15 epochs our learning rate drops, creating the “stair-step”-like effect.

Figure 5 (proper) demonstrates the basic signs of step-based learning rate scheduling — you possibly can clearly see our:

  1. Coaching/validation loss decrease
  2. Coaching/validation accuracy improve

…when our learning rate is dropped.

This is especially pronounced in the first two drops (epochs 15 and 30), after which the drops turn into much less substantial.

This sort of steep drop is a basic sign of a step-based learning rate schedule being utilized — should you see that sort of training conduct in a paper, publication, or one other tutorial, you could be virtually positive that they used step-based decay!

Getting back to our accuracy, we’re now at 86-87% accuracy, an improvement from our first experiment.

Experiment #four: Linear learning rate schedule results

Let’s attempt utilizing a linear learning rate schedule with Keras by setting 
power=1.0 :

Determine 6: Linear learning rate decay (left) utilized to ResNet on CIFAR-10 over 100 epochs with Keras. The coaching accuracy/loss curve is displayed on the proper.

Figure 6 (left) exhibits that our learning rate is reducing linearly over time while Figure 6 (right) visualizes our training historical past.

We’re now seeing a sharper drop in each training and validation loss, particularly previous roughly epoch 75; nevertheless, observe that our coaching loss is dropping considerably quicker than our validation loss — we may be vulnerable to overfitting.

Regardless, we at the moment are acquiring 88% accuracy on our knowledge, our greatest outcome so far.

Experiment #5: Polynomial learning rate schedule outcomes

As a remaining experiment let’s apply polynomial learning rate scheduling with Keras by setting
power=5 :

Figure 7: Polynomial-based learning decay outcomes using Keras.

Figure 7 (left) visualizes the truth that our learning rate is now decaying in line with our polynomial perform whereas Determine 7 (right) plots our coaching historical past.

This time we get hold of ~86% accuracy.

Commentary on learning rate schedule experiments

Our best experiment was from our fourth experiment where we utilized a linear learning rate schedule.

However does that mean we should always all the time use a linear learning rate schedule?

No, removed from it, truly.

The important thing takeaway right here is that for this:

  • Specific dataset (CIFAR-10)
  • Specific neural community architecture (ResNet)
  • Initial learning rate of 1e-2
  • Number of training epochs (100)

…is that linear learning rate scheduling labored one of the best.

No two deep learning tasks are alike so you’ll need to run your personal set of experiments, together with various the preliminary learning rate and the entire number of epochs, to find out the suitable learning rate schedule (further commentary is included in the “Summary” section of this tutorial as properly).

Do different learning rate schedules exist?

Different learning rate schedules exist, and actually, any mathematical perform that can accept an epoch or batch number as an enter and returns a learning rate may be thought-about a “learning rate schedule”. Two other learning rate schedules chances are you’ll encounter embrace (1) exponential learning rate decay, in addition to (2) cyclical learning rates.

I don’t typically use exponential decay as I discover that linear and polynomial decay are more than enough, however you’re more than welcome to subclass the
LearningRateDecay  class and implement exponential decay should you so want.

Cyclical learning rates, however, are very highly effective — we’ll be masking cyclical learning rates in a tutorial later on this collection.

How do I select my initial learning rate?

You’ll discover that on this tutorial we didn’t differ our learning rate, we stored it fixed at
1e-2 .

When performing your personal experiments you’ll need to combine:

  1. Learning rate schedules…
  2. …with totally different learning rates

Don’t be afraid to combine and match!

The 4 most necessary hyperparameters you’ll need to discover, embrace:

  1. Preliminary learning rate
  2. Number of training epochs
  3. Learning rate schedule
  4. Regularization power/quantity (L2, dropout, and so on.)

Discovering an applicable stability of every could be difficult, however by means of many experiments, you’ll have the ability to discover a recipe that leads to a extremely correct neural community.

In the event you’d wish to study extra about my ideas, strategies, and greatest practices for learning charges, learning rate schedules, and training your personal neural networks, confer with my e-book, Deep Learning for Pc Vision with Python.

The place can I study extra?

At this time’s tutorial launched you to learning rate decay and schedulers using Keras. To study more about learning rates, schedulers, and the right way to write custom callback features, seek advice from my e-book, Deep Learning for Pc Vision with Python.

Inside the guide I cowl:

  1. More details on learning charges (and how a strong understanding of the idea impacts your deep learning success)
  2. Methods to spot beneath/overfitting on-the-fly with a custom training monitor callback
  3. Learn how to checkpoint your models with a customized callback
  4. My ideas/tips, recommendations, and greatest practices for training CNNs

In addition to content material on learning rates, you’ll also find:

  • Super sensible walkthroughs that current options to actual, real-world image classification, object detection, and occasion segmentation issues.
  • Palms-on tutorials (with a lot of code) that not solely present you the algorithms behind deep learning for pc vision but their implementations as nicely.
  • A no-nonsense educating type that’s assured that will help you grasp deep learning for picture understanding and visible recognition.

To study extra concerning the guide, and seize the table of contents + free sample chapters, simply click on right here!


On this tutorial, you discovered easy methods to make the most of Keras for learning rate decay and learning rate scheduling.

Particularly, you discovered find out how to implement and make the most of a variety of learning rate schedules with Keras, including:

  • The decay schedule built into most Keras optimizers
  • Step-based learning rate schedules
  • Linear learning rate decay
  • Polynomial learning rate schedules

After implementing our learning rate schedules we evaluated each on a set of experiments on the CIFAR-10 dataset.

Our results demonstrated that for an preliminary learning rate of
1e-2 , the linear learning rate schedule, decaying over
100  epochs, carried out the perfect.

Nevertheless, this does not mean that a linear learning rate schedule will all the time outperform other varieties of schedules. As an alternative, all this implies is that for this:

  • Specific dataset (CIFAR-10)
  • Specific neural network structure (ResNet)
  • Preliminary learning rate of
  • Variety of coaching epochs (
    100 )

…that linear learning rate scheduling worked the most effective.

No two deep learning tasks are alike so you’ll need to run your personal set of experiments, together with various the initial learning rate, to determine the appropriate learning rate schedule.

I recommend you keep an experiment log that details any hyperparameter decisions and related outcomes, that means you possibly can refer again to it and double-down on experiments that look promising.

Don’t anticipate that you simply’ll be capable of practice a neural network and be “one and done” — that not often, if ever, occurs. As an alternative, set the expectation with your self that you simply’ll be operating many experiments and tuning hyperparameters as you go along. Machine learning, deep learning, and artificial intelligence as an entire are iterative — you build in your earlier results.

Later in this collection of tutorials I’ll also be displaying you learn how to choose your initial learning rate.

To obtain the supply code to this submit, and be notified when future tutorials are revealed here on PyImageSearch, just enter your e mail handle in the type under!