Tutorial on Algorithm Configuration (PPSN'20)

Website: automl.org/events/tutorial_ac_2020/
Slides: automl.org/wp-content/uploads/2020/08/AC_Tutori…
Colab: colab.research.google.com/drive/1v0ZH5S9Sfift30…

In particular for algorithms in AI (incl. machine learning, deep learning, evolutionary algorithms, and hard-combinatorial problem solvers, such as SAT solving), it is crucial to determine a good configuration of parameter settings to achieve peak performance. Since an in-depth understanding of these high-dimensional, abstract black-box problems is often not available, manual tuning of parameter configurations is often considered as a tedious, time-consuming and even error-prone task. Therefore, automated algorithm configuration (AC) systems were proposed that free users from this task, and they demonstrated the practical applicability of these methods by achieving substantial speedups in a wide range of applications (e.g., up to 3,000x in SAT, 50x in MIP, 300x in AI planning and 14x in ASP).

To achieve this, traditional AC systems first optimize the configuration on a set of problem instances wrt an arbitrary performance metric, and henceforth the found configurations will be applied to new instances. Although applying any kind of optimization approach (incl. evolutionary algorithms and model-based optimization) to AC might seem to be straightforward, AC systems are faced with many challenges in practice, incl. (i) configuration spaces might contain categorical and continuous parameters with hierarchical dependencies (ii) a single evaluation takes a lot of time, (iii) it is often infeasible to evaluate many configurations on all instances and thus (iv) we might only partially observe censored performance estimates, since we cannot afford to run all evaluations until the end. In the tutorial, we will give an overview on (i) how to address these problems, (ii) how recent advances in the field led to even more efficient systems than ever before, (iii) how to apply these to different applications, including applications to automated machine learning (AutoML) and SAT solving, and (iv) open challenges in the field.