Hyperparameter Optimization in NN

  • Hyperparameter tuning often leads to huge performance gain
  • Too big space to optimize by hands
  • How should we go about automating hyperparameter optimization?

Grid Search vs. Random Search

Figure 1
  • Try various hyperparameter settings and take the best one!
  • Can we do better than this? → Yes, apply bayesian optimization.

Bayesian Optimization

  • We need two design choices
  • Surrogate Modelling Function — Gaussian Processes
  • Acquisition Function — Probability of Improvement, Expected Improvement

Example

Joonsu Oh

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store