Jenkins World 2017

Last year our company’s internal Jenkins instance had become overwhelmed, as we had grown dramatically over the last few years. It contained jobs from all our teams and departments, we had too many administrators, we wanted to improve change control practice and the plugin upgrade process since the latter could affect more people than just the team upgrading a plugin. This was all a result of the rapid business growth at CloudBees. The tipping point came when we wanted to do even more testing of our products which now led to long queue times for simple jobs.

So the decision was made to move off this single instance to a new cluster,where each team could have its own master and, using Docker, gain greater control over its build environment.

Rather than migrating what we had, we took the opportunity to change and improve our teams’ setup and dogfood many of the shiny new features that were in development at the time. Taking the approximately 150 jobs and replacing them with declarative pipelines configured automatically from GitHub using custom marker files (a proprietary feature) we now no longer have to create jobs for new plugins. This process is now performed automatically. We even have a way for an individual plugin to replace the default pipeline - all controlled via the SCM.

However things were not clear sailing: Dogfooding caused us to find issues, wanting to accelerate sometimes pushed us to simply reconsider some of our existing processes instead of automating them.

Having a way to share a single pipeline across projects is great, but it also obviously raises the criticality of it. Initially, we were sometimes breaking all our jobs because we had to manually test the default pipeline when changing it and it was possible to miss a mistake. Due to these complexities, we had to work out how to test that "default pipeline," as well as devise a workable strategy that would enable us to reach our end goal of continuous delivery.

This presentation will share our experiences and techniques for dealing with a similar migration.