Parallel Firehose

Roy Osherove ·

Pattern: Parallel Firehose

Other Names:

  • Build Environment Per Stakeholder

Symptoms:

You can't deploy a new version of the application to your test or staging environment because it's in use by other people in the organization such as the QA team, Product owners, or customers. So you can only deploy during the night time, or some other constraint.

Problem:

Developers are sharing the same environment (machines, physical or virtual) with other stakeholders.

Forces:

  • You want to do continuous delivery, without needing to wait for others.
  • Other stakeholders want to use or test the application without needing to stop, or be annoyed by your deployments making the application slower or crashing.

Solution:

Create a separate environment for your other stakeholders, that they can use without you interfering with it, and they can pull a new version to at will.

Example:

Assuming you have the following environment:

  • A Build Machine that runs TeamCity or Jenkins
  • A Build Agent Machine (Build1) that runs a TeamCity agent, or a Jenkins Worker
  • A Test Machine (Test1) where BUILD1 deploys your application to for demo purposes and for use by QA
  • A developer machine (DEV1) that keeps checking in code to the main branch

You want to do continuous testing by running unit and integration tests on check in, as well as smoke tests and acceptance tests by deploying the application automatically on every check in.

You can't because QA are using the machine during work hours. You can only deploy and run those tests at night, and they might actually break whatever nightly tests the QA folks are running.

Here is one way to solve this:

Setup the environment with new machines so you end up with an environment like this:

  • A Build Machine (BUILDMASTER) that runs TeamCity or Jenkins
  • A Build Agent Machine (Build1) that runs a TeamCity agent, or a Jenkins Worker
  • A Build Agent Machine (Build2) that runs a TeamCity agent, or a Jenkins Worker
  • A Test Machine (Test1) that an agent deploys your application to for dev use
  • A Test Machine (Test2) that an agent deploys your application to for QA use
  • A developer machine (DEV1) that keeps checking in code to the main branch

Then, in your CI server setup the following two workflows:

workflow 1: Fully automated Pipeline

  • Triggered on Check in to main branch
  • compile on a free agent
  • Run unit tests on a free agent
  • Run Integration tests on a free agent
  • Deploy to TEST1 on a free agent
  • Run acceptance tests only on TEST1

workflow 2: Pull Based QA Pipeline

  • Triggered by QA clicking on a button, or on a set schedule, or both
  • compile using a free agent
  • Run unit tests using a free agent
  • Run Integration tests using a free agent
  • Deploy to TEST2 using a free agent
  • run QA tests on TEST2

Summary

With this approach nobody is stepping on anyone's toes:

  • Developers can do continuous tests on TEST1 dozens of times a day (yes, some teams actually achieve this). A.k.a it's so fast it's like "drinking from a fire hose" (hence the pattern name)
  • QA get a nice stable environment they can control and pull versions into.

You can repeat this process for new stakeholders. If you're using cloud services, it's even easier to setup such new environments.