Review: PR-080-Practical Bayesian Optimization of Machine Learning
Hyperparameter Optimization in NN
- Hyperparameter tuning often leads to huge performance gain
- Too big space to optimize by hands
- How should we go about automating hyperparameter optimization?
Grid Search vs. Random Search
Figure 1- Try various hyperparameter settings and take the best one!
- Can we do better than this? → Yes, apply bayesian optimization.
Bayesian Optimization
- We need two design choices
- Surrogate Modelling Function — Gaussian Processes
- Acquisition Function — Probability of Improvement, Expected Improvement
Example
Figure 2Figure 3Figure 4Figure 5Figure 6
