Continuous Integration has never been so easy

Running tests in a shell step

Script used to run tests

Setup

1. Install Jenkins CI in a server
2. Listen to repository changes with GitHub Pull Request Builder or Generic Webhook Trigger
3. Set up the job(s)

Pros: free and customisable

Install it, create the jobs, and adapt them to your team's workflow

Pros: temporary testing environments

Cons

• Lot of setup, tweaking, and server maintenance required
• It's tricky to keep the job implementation under version control

The big con: software is free, time isn't

• As a self-hosted option, we dive into a real case of the costs of maintaining CI tooling with Jenkins

Assumptions

• Eight developers
• 10+ repositories
• 4 jobs per pull request, 18 minutes total CPU time
• 8 PRs / day, testing 24 total commits
• Team runs some tests locally, reducing CI load due to reworked tests

Engineering costs

Juampy spent:

• 116 hours over 15 months
• Time entries logged with keywords Jenkins, CI, or Continous (no migrations!)
• Literally hundreds to thousands of dollars per month
• Doesn't include client team hours which was all of the infrastructure and hosting

Sample

Open it at GitHub

              unit_kernel_tests: &unit_kernel_tests
  <<: *defaults
  steps:
    - checkout
    - *copy_robo
    - restore_cache: *restore_cache
    - run:
        name: Run PHPUnit tests
        command: robo job:run-unit-tests
    - store_test_results:
        path: /var/www/html/artifacts/phpunit
    - store_artifacts:
        path: /var/www/html/artifacts
    - save_cache: *save_cache
              
            

Setup

1. Add a .circleci/config.yml file to the repository
2. Authenticate with your GitHub account at https://circleci.com
3. Allow CircleCI to watch for repository changes

Pros: docker-compose-style environments

              defaults: &defaults
  docker:
    - image: juampynr/drupal8ci:latest

    - image: selenium/standalone-chrome-debug:3.7.1-beryllium

    - image: mariadb:10.3
      environment:
        MYSQL_ALLOW_EMPTY_PASSWORD: 1
              
            

Pros: SSH access to job environments

Pros: Run jobs locally

Cons

• The free service only allows one concurrent job at a time
• Not as customizable as a Jenkins setup

Pros: Costs

• ~50 hours from "we have never used Circle" to done
• Very minimal maintenance - a few hours twice per year for Drupal core upgrades
• Free to ~250 USD / month, scales with your org

.travis.yml

View full file

              cache:
  directories:
    - $HOME/.composer/cache/files
services:
  - docker
env:
  matrix:
    - JOB=job:check-coding-standards
    - JOB=job:run-unit-tests
    - JOB=job:run-behat-tests
script:
  - vendor/bin/robo $JOB
              
            

Cons

• Coverage reports require a third party service like Coveralls
• Supports Docker but needs extra setup
• No Command Line Interface to run jobs locally

But at what cost?

Developers

"Tests are too hard"

Tests are tools and APIs that teach you the underlying systems.

QA

"We don't trust the tests"

If you have a dedicated QA team, they need to be completely integrated with development.

Project Managers

"We will spend more time on tests than fixing bugs"

Be tactical about what to test. Trust your developers. Track your hours.

Product Owners

"We need to deliver"

Continuous integration leads to continuous delivery.