Understanding the performance of different databases is crucial for building efficient data-intensive applications.
Benchmarking possible databases is a valid option for enabling such comparison, yet, it is a major challenge due to heterogeneity on the resource, database and workload level.
To enable an on-demand comparison of cloud-hosted databases we present a database performance ranking as an open dataset that enables the performance and cost comparison in a multi cloud database and workload context.

In our talk we are discussing the challenges for carrying out transparent, reproducible and comparable benchmarking case studies for cloud-hosted database systems. In particular, we discuss impact factors on the cloud resource and database level that need to be considered for fair comparisons.

Based on our open database performance ranking, we demonstrate what kind of data is required to enable transparent, reproducible and comparable benchmarking results and also discuss the insights from large-scale benchmarking studies of cloud-hosted database systems.

This enables the audience to take away the following lessons-learned:
– what is required to enable transparent and pitfalls in running large-scale benchmarking studies in the cloud
– performance insights of different database systems
– performance insights of different cloud providers
– performance/cost comparisons of databases operated on IaaS vs. DBaaS
– performance insight for database systems under different workloads