Assessing irace for automated machine learning

Abstract

Automated algorithm engineering tools have become an important asset for both academia and industry. In general, these tools are powered by a few, provenly effective algorithm configurators, among which is irace. In this proof-of-concept investigation, we assess the application of irace to the field of machine learning (ML). To do so, we propose a template built on top of the scikit-learn algorithmic framework, dubbed isklearn, comprising many preprocessing, feature engineering, and prediction algorithms. Furthermore, we formally define a configuration space and an experimental setup that allow irace to treat machine learning datasets as instances of an optimization problem, making isklearn a fully functional automated machine learning (AutoML) system. Preliminary results demonstrate that irace is able to engineer effective algorithms for three of the major ML application domains, namely computer vision, natural language processing, and time series analysis.