AC

Algorithm Configuration

Let \(\mathcal{I}\) be the set of problem instances, \(\mathcal{D}\) be the probability distribution over \(\mathcal{I}\). Let \(\mathcal{A}\) be the set of algorithms with parameters \(p_1, p_2, \ldots, p_k\). The domain of each parameter \(p_i\) is denoted as \(\Theta_i\), and the parameter space is defined as \(\Theta = \Theta_1 \times \Theta_2 \times \ldots \times \Theta_k\).

The objective of algorithm configuration is to find the optimal parameter configuration \(\theta^* \in \Theta\) that minimizes the expected cost of the algorithm over the distribution of problem instances:

\[ \theta^* \in \arg\min_{\theta \in \Theta} \mathbb{E}_{i \sim \mathcal{D}}[c(\theta, i)] \]

where \(c(\theta, i)\) is the cost of running algorithm \(\mathcal{A}\) with parameter configuration \(\theta\) on problem instance \(i\).

Dynamic Algorithm Configuration

Hyperparameter Optimization in Reinforcement Learning

Eimer et al. (2023) discussed the hyperparameter optimization problem in the context of reinforcement learning (RL).¹ They conducted a experimental study on the performance of various hyperparameter optimization methods, including random search, DEHB (Awad et al. 2021),² and BGT (Wan et al. 2022).³

• Optuna (Akiba et al. 2019)

References

Akiba, Takuya, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. “Optuna: A Next-Generation Hyperparameter Optimization Framework.” Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (New York, NY, USA), 2623–31. https://doi.org/10.1145/3292500.3330701.

Awad, Noor, Neeratyoy Mallik, and Frank Hutter. 2021. “DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization.” Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (California). https://doi.org/10.24963/ijcai.2021/296.

Eimer, Theresa, Marius Lindauer, and Roberta Raileanu. 2023. “Hyperparameters in Reinforcement Learning and How to Tune Them.” arXiv [Cs.LG], ahead of print. https://doi.org/10.48550/arXiv.2306.01324.

Wan, Xingchen, Cong Lu, Jack Parker-Holder, et al. 2022. “Bayesian Generational Population-Based Training.” arXiv [Cs.LG], ahead of print. https://doi.org/10.48550/arXiv.2207.09405.

Footnotes

1. Source code: https://github.com/facebookresearch/how-to-autorl.

2. Source code: https://github.com/automl/DEHB.

3. Source code: https://github.com/xingchenwan/bgpbt.