Background
Over the past few years, we have been using GitHub for storing the code of our private and open-source projects.
We also have a Continuous Integration (CI) solution in place, to ensure that every new piece of code we write doesn't break anything in any of our applications. This solution was made out of three parts:
Part 1: Jenkins
We used Jenkins to launch our CI builds at every git push to a branch (master or another). This was an instance that we managed ourselves.
Part 2: Janky
We used Janky to make the communication between GitHub and Jenkins possible: GitHub would forward events to Janky, and Janky would in turn forward these to Jenkins (and vice-versa). We also had some logic there to setup new projects via some XML templates. This was an instance that we also had in-house.
Part 3: Babysteps
We are always trying to have one-liner commands to do things, and releasing a new Debian package or gem version is no exception. We used Babysteps to manage application releases automatically. It is built-in with a REST API that our Slack bot triggered.
Why have we changed?
Over time, we encountered some issues with that CI solution that made us want to change it:
Flaky CI process
The GitHub / Janky / Jenkins suite was sometimes not triggering jobs in Jenkins, or triggering jobs but not notifying Janky about it, etc. This made us do some unnecessary and sometimes time-consuming debugging for something that we wanted to just work.
CI/CD Pipelines support in Jenkins
Although Jenkins made a lot of progress in their Pipeline support, it still felt complex and awkward to use for our cases. We wanted a solution similar to the ones present in any modern CI solution: one yaml file for each project that describes a pipeline.
It was getting old
That CI solution has been around for the past 5 years, and we also felt like it was not standing up to our new requirements. We felt it was time to let it go.
Why not a cloud-based solution?
Mostly because of the pricing and the number of projects we have currently (over 150). We also have the knowledge to build our in-house CI solution without too much pain, so we decided to take advantage of that.
Our new setup
For our private projects, we chose GitLab as a replacement for GitHub and our CI solution. It is a well-oiled monolith that is extremely easy to install, upgrade and customize. In GitLab, pipelines and docker are both first-class citizen, which is exactly what we were missing with our previous CI solution.
Furthermore, the core team is making a new release every 22nd of each month and patches in between. This is simply first-class maintenance to us. On top of all this, it is open source project, which allowed us to have our in-house instance #TakeThatNSA.
Our experience so far
Smooth transition
With the help of GitLab project importer, importing all our projects from GitHub took around 30 minutes. It even took all of our Pull Requests and releases/tags/comments!
We then had to add a gitlab-ci.yml per project, depending on its type (Debian package, web application, gem, ...). We made a small script to tag our projects and to upload the desired gitlab-ci.yml to it. What made that migration quite smooth was the fact that we use docker-compose for our development/test environments.
We have set up a GitLab runner with docker-compose installed as a dependency, and we were good to go!
Simpler workflows
To release a new version of a ruby gem or a Debian package, we now need only one command for our Slack bot: lita release <project name>. The whole process is then done as a GitLab pipeline: Testing the application, building the gem/package and pushing it to our internal repository.
All of the pipeline information is encapsulated inside the gitlab-ci.yml, which makes our processes extremely flexible: ruby app with docker-compose for tests, go application that will be compiled in a Debian package, web application deployed to Openshift: all of this possible without breaking a sweat.
Conclusion
Until now, our CI has been very stable and predictable: it finally just works®. We are very happy about the change and we believe that it will allow to deliver value to our customers even more frequently with increased confidence.
