Chronos: The Swiss Army Knife for Database Evaluations

Authors
Marco Vogt, Alexander Stiemer, Sein Coray, Heiko Schuldt
Type
In Proceedings
Date
2020/4
Appears in
Proceedings of the 23rd International Conference on Extending Database Technology (EDBT)
Location
Copenhagen, Denmark
Abstract

Systems evaluations are an important part of empirical research in computer science. Such evaluations encompass the systematic assessment of the run-time characteristics of systems based on one or several parameters. Considering all possible parameter settings is often a very tedious and time-consuming task with many manual activities, or at least the manual creation of evaluation scripts. Ideally, the thorough evaluation of a complete evaluation space can be fully automated. This includes the set-up of the evaluation, its execution, and the subsequent analysis of the results. In this paper, we introduce Chronos, a system for the automation of the entire evaluation workflow. While Chronos has originally been built for database systems evaluations, its generic approach also allows its usage in other domains. We show how Chronos can be deployed for a concrete database evaluation, the comparative performance analysis of different storage engines in MongoDB.