When trying to use CLR, the LR range test in the original paper was a little confusing for me. Also, I couldn't reproduce the kind of graph the author got after running the LR range test.
I managed to get a graph like the one in fast.ai after a lot of experimentation.
The paramters of the LR range test that worked for me were:
clr = CyclicLR(base_lr=0.00001, max_lr=1.5, step_size=5000)
BATCH_SIZE=1024
history_init = tpu_model64.fit(
X_train, y_train_onehot, batch_size=BATCH_SIZE,
epochs=50,
callbacks=[clr]
)
Plotting clr.history['loss'] against clr.history['lr'] gave me a nice smooth graph to choose the LR from:
