Improving and extending the benchmark suite for Polypheny (Master Project, Ongoing)
Author
Description
Benchmarking database management systems (DBMSs) is not simple. There are many
factors involved in choosing a benchmark, depending on the data model and intended use
case for the DBMS (represented by the workload). This becomes even more challenging
when benchmarking PolyDBMSs. PolyDBMSs, by design, support multiple data models
and are intended to provide acceptable performance across a wide variety of workloads.
Further PolyDBMSs support different underlying data stores, which again influence the
results of the benchmark. Because of that a single benchmark is not enough for PolyDBMSs.
To efficiently benchmark PolyDBMSs, multiple benchmarks have to be used.
Currently, benchmarking for the PolyDBMS Polypheny is done with the Simple Client1 .
The Simple Client is written in Java and uses Chronos [Vog20] to retrieve benchmark
jobs and report the results. The Simple Client implements some benchmarks, but also
supports executing benchmarks from the relational benchmark suite OLTPBench [Dif13].
OLTPBench in turn implements various relational benchmarks, including the standard-
ized TPC-C and TPC-H benchmarks. Unfortunately, OLTPBench is no longer actively
maintained and has been superseded by the BenchBase suite. The Simple Client uses a
fork of OLTPBench with added Polypheny support.
The Simple Client runs OLTPBench and Polypheny directly on the host without isolation.
This makes cleaning up after benchmark runs difficult.
The goal of this project is to add new external benchmarks to the Simple Client for
document oriented workloads, such as YCSB and CH2 [Coo10; Car22]. This also requires
adding Polypheny support to those benchmarks, and potentially, extending document
support inside Polypheny. The overall process to add new external benchmarks to the
Simple Client should be improved as well. The external benchmarks should be patched
automatically, when possible, without the need of maintaining forks. Also the Simple
Client should execute the benchmarks and Polypheny inside containers to improve isolation
and cleanup.
If time permits, OLTPBench should be replaced by the maintained BenchBase.
Objectives
• Add YCSB and CH2 as external benchmarks to the Simple Client.
• Execute external benchmarks inside containers.
• Streamline the process to add new external benchmarks to the Simple Client.
• Verify the implementation for correctness and robustness.
Optional objectives
• Replace OLTPBench by BenchBase.
• Automatically patch/extend external benchmarks to support Polypheny.
• Improve running benchmarks locally.
Since Polypheny is licensed under the Apache 2.0 license, the developed solution has to be
licensed under the Apache 2.0 license, too.
Start / End Dates
2026/09/01 - 2026/12/01