CIPerf: a benchmark for continuous integration services performance and cost analysis

Abstract

Continuous Integration is a crucial practice in modern software development, enabling teams to automate the process of building, testing, and merging code increments to ensure continuous delivery of high-quality software. Despite its growing adoption, the cost and performance of Continuous Integration services often go unexamined in sufficient detail. This paper presents CIPerf, a comprehensive benchmark designed to analyze both the performance and cost of cloud-based and self-hosted Continuous Integration services. The study centers on a comparison between two specific services: Bitbucket Pipelines, a cloud-based offering by Atlassian, and Hetzner, a self-hosted solution. By focusing on these platforms, the research aims to provide practical insights into the real-world costs and execution performance of Continuous Integration services. To achieve this, CIPerf conducted automated tests on an hourly basis over a two-month period, measuring critical timeframes such as resource provisioning, environment setup, and the actual test execution times. The results showed significant differences in both the cost efficiency and the consistency of performance between the two services. For instance, Bitbucket Pipelines, while convenient in its cloud-based offering, demonstrated higher variability in provisioning times compared to the stable, predictable performance of Hetzner’s self-hosted environment. The analysis also explored how these performance metrics influence key software development metrics, including deployment frequency and developer productivity. CIPerf provides a clear methodology for developers and organizations to objectively assess their Continuous Integration service options, ultimately helping them optimize their workflows. Moreover, this benchmark can serve as an ongoing tool to monitor service performance over time, identifying potential degradations or improvements in service quality, thus offering long-term value for teams that rely on Continuous Integration for their development processes.