Cyclical Learning Rate

When trying to use CLR, the LR range test in the original paper was a little confusing for me. Also, I couldn't reproduce the kind of graph the author got after running the LR range test.

I managed to get a graph like the one in after a lot of experimentation.

The paramters of the LR range test that worked for me were:

clr = CyclicLR(base_lr=0.00001, max_lr=1.5, step_size=5000)


history_init =
    X_train, y_train_onehot, batch_size=BATCH_SIZE,

Plotting clr.history['loss'] against clr.history['lr'] gave me a nice smooth graph to choose the LR from: