CIPerf: a benchmark for continuous integration services performance and cost analysis

Abstract

Continuous Integration is a crucial practice in modern software development, enabling teams to automate the process of
building, testing, and merging code increments to ensure continuous delivery of high-quality software. Despite its growing adoption,
the cost and performance of Continuous Integration services often go unexamined in sufficient detail. This paper presents CIPerf, a
comprehensive benchmark designed to analyze both the performance and cost of cloud-based and self-hosted Continuous Integration
services. The study centers on a comparison between two specific services: Bitbucket Pipelines, a cloud-based offering by Atlassian,
and Hetzner, a self-hosted solution. By focusing on these platforms, the research aims to provide practical insights into the real-world
costs and execution performance of Continuous Integration services. To achieve this, CIPerf conducted automated tests on an hourly
basis over a two-month period, measuring critical timeframes such as resource provisioning, environment setup, and the actual test
execution times. The results showed significant differences in both the cost efficiency and the consistency of performance between
the two services. For instance, Bitbucket Pipelines, while convenient in its cloud-based offering, demonstrated higher variability in
provisioning times compared to the stable, predictable performance of Hetzner’s self-hosted environment. The analysis also explored
how these performance metrics influence key software development metrics, including deployment frequency and developer
productivity. CIPerf provides a clear methodology for developers and organizations to objectively assess their Continuous Integration
service options, ultimately helping them optimize their workflows. Moreover, this benchmark can serve as an ongoing tool to
monitor service performance over time, identifying potential degradations or improvements in service quality, thus offering longterm value for teams that rely on Continuous Integration for their development processes.