🇬🇧 English | 🇰🇷 한국어

Practical comparison of 5 hyperparameter optimization algorithms for machine learning

🇰🇷 한국어 | 🇬🇧 English | 🎯 Quick Start | 📊 Results

Typical Performance Pattern (Diabetes Dataset: 442 samples, 10 features, 50 iterations)

💡 Important Note: Actual results vary based on random_state, data split, and environment. All methods typically improve baseline by 20-27%. Run the notebook to see results on your machine.

⚡ Key Insight: TPE (Hyperopt) achieved the highest improvement (+27.12%), closely followed by Random Search (+26.33%) and Optuna (+26.02%). Modern Bayesian methods consistently outperform Grid Search with better efficiency.

1. Grid Search - Exhaustive search through all parameter combinations
2. Random Search - Random sampling from parameter distributions
3. Optuna - Modern TPE with pruning (replaces deprecated HyperBand)
4. Bayesian Optimization - Probabilistic model-based optimization
5. TPE (Hyperopt) - Tree-structured Parzen Estimator

• Understand strengths and weaknesses of each algorithm
• Know which method to choose for different scenarios
• Implement optimization in real projects with working code
• Compare results with statistical rigor
• Reduce hyperparameter tuning time significantly

📌 Click to view Hyperparameter Optimization Concept Mind Map

git clone https://github.com/hyeonsangjeon/Hyperparameters-Optimization.git
cd Hyperparameters-Optimization
pip install -r requirements.txt

Interactive Notebook (Recommended)

jupyter notebook HyperParameterInspect.ipynb        # Korean
jupyter notebook HyperParameterInspect_EN.ipynb     # English

Automated Benchmark

python benchmark_hpo_algorithms.py

• Dataset: Sklearn Diabetes (442 samples, 10 features)
• Model: LightGBM Regressor
• Iterations: 50 trials per method
• Validation: 2-fold cross-validation
• Metric: Mean Squared Error (lower is better)

⚠️ Note: Values shown are from recent benchmark run. Absolute MSE values vary by environment and random_state, but the ranking and relative performance are consistent across runs.

Hyperparameters-Optimization/
├── HyperParameterInspect.ipynb           # Korean tutorial notebook
├── HyperParameterInspect_EN.ipynb        # English tutorial notebook
├── benchmark_hpo_algorithms.py           # Automated benchmark script
├── requirements.txt                      # Python dependencies
├── README.md                             # This file
├── pic/                                  # Images and plots
└── doc/                                  # Additional documentation

Core Dependencies

• Python 3.8+
• numpy, pandas, scikit-learn, lightgbm

Optimization Libraries

• optuna >= 3.0.0 (Modern HPO with pruning)
• hyperopt >= 0.2.7 (TPE algorithm)
• scikit-optimize >= 0.9.0 (Bayesian optimization)

Visualization

• matplotlib, jupyter

⚠️ Important: This project uses Optuna instead of the deprecated scikit-hyperband library due to compatibility issues with modern scikit-learn versions.

• Random Search: Bergstra & Bengio, JMLR 2012
• TPE: Bergstra et al., NIPS 2011
• Bayesian Optimization: Snoek et al., 2012
• HyperBand: Li et al., ICLR 2018

• 🎤 Hyeonsang Jeon, "Expert Lecture: Hyperparameter Optimization in AI Modeling", ITDAILY, 2022. Article

• 🎤 Hyeonsang Jeon, "Case Study: AutoDL with Hyperparameter Optimization in Deep Learning Platforms", AI Innovation 2020, The Electronic Times, 2020. Video

• 📰 Featured in ComWorld

Contributions welcome! Ways to help:

• 🐛 Report bugs or issues
• 💡 Suggest new features or algorithms
• 📝 Improve documentation
• 🌍 Translate to other languages
• 🔬 Add optimization methods

Development Setup

git clone https://github.com/YOUR_USERNAME/Hyperparameters-Optimization.git
cd Hyperparameters-Optimization
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python benchmark_hpo_algorithms.py

MIT License - see LICENSE file for details.

Hyeonsang Jeon
GitHub: @hyeonsangjeon

Special thanks to:

• Optuna - Modern HPO framework
• Hyperopt - TPE implementation
• scikit-optimize - Bayesian optimization
• LightGBM - Fast gradient boosting

• Optuna - Next-generation HPO framework
• Hyperopt - Distributed HPO library
• scikit-optimize - Bayesian optimization
• Ray Tune - Scalable distributed tuning