Hyperparameter Optimization in NN

  • Hyperparameter tuning often leads to huge performance gain
  • Too big space to optimize by hands
  • How should we go about automating hyperparameter optimization?

Grid Search vs. Random Search

Figure 1
  • Try various hyperparameter settings and take the best one!
  • Can we do better than this? → Yes, apply bayesian optimization.

Bayesian Optimization

  • We need two design choices
  • Surrogate Modelling Function — Gaussian Processes
  • Acquisition Function — Probability of Improvement, Expected Improvement

Example

  • It takes a lot of space to store NN weights and memory to infer from it.
  • How can we make this situation better? → Model Compression
Figure 1
  • Exactly what kind of model compression technique should we use?
  • This paper can be an answer to one of them.

Pruning

Joonsu Oh

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store