# Pipeline Driven: full text > Pipeline Driven is Roy Osherove's site about continuous delivery: build pipelines, pipeline-friendly testing, DevOps metrics and enterprise DevOps. Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/. Source: https://pipelinedriven.org/ . Generated 2026-10-23T00:45:37.694Z. 41 posts, 27 pages. ## About the author **Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: https://x.com/royosherove - LinkedIn: https://www.linkedin.com/in/osherove/ ### Contact - Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7 ### License Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/ --- title: "A Metrics Framework for Continuous Delivery" url: "https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery.md" canonical: "https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery" published: "2022-07-11T17:05:14.945000+00:00" updated: "2022-07-11T17:24:28.084000+00:00" tags: [] author: "Roy Osherove" description: "Here’s a framework I like to start with when I discuss what types of metrics can help or harm you in your journey to continuous delivery: Lagging Indicators…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # A Metrics Framework for Continuous Delivery 11 July 2022 · 3 min read · Roy Osherove Here’s a framework I like to start with when I discuss what types of metrics can help or harm you in your journey to continuous delivery: [![Metrics Framework for continuous delivery](https://pipelinedriven.org/img/c/metrics-framework-3f808b.jpg)](https://pipelinedriven.org/img/c/metrics-framework-3f808b.jpg "View full size") ## Lagging Indicators At the organizational level, unit tests are usually part of a bigger set of goals, usually related to continuous delivery. If that's the case for you, I highly recommend using the four common DevOps metrics : • Deployment Frequency—How often an organization successfully releases to production • Lead Time for Changes— o The amount of time it takes a feature request to get into production NOTE: Many places incorrectly publish this as : The amount of time it takes a commit to get into production – which is only a part of the journey a feature goes through from an organizational standpoint. If you're measuring from commit time – you're closer to measuring the "cycle time" of a feature from commit up to a specific point. Lead time is made up of multiple cycle times. • Change Failure Rate— Number of failures found in production per release/Deployment/time OR: The percentage of deployments causing a failure in production OR • Time to Restore Service—How long it takes an organization to recover from a failure in production These four are we'd call "Lagging Indicators" and are very hard to fake (although pretty easy to measure in most places). They are great in making sure we do not lie to ourselves about the results of experiments. ## Leading Indicators However, many times we'd like a faster feedback loop that we're going ther right way, that's where "leading indicators" come in. Leading indicators are things we can control on the day to day view – code coverage, number of tests, build run time and more. They are easier to "fake" but, combined with lagging indicators, can often provide us with early signs we might be going the right way. We can measure leading indicators at the team and middle management level, but we have to make sure to always have the lagging indicators as well, so we do not lie to ourselves that we’re doing well when we’re not. ## Metric groups and categories I usually break up the leading indicators to two types of groups: ·        **Team Level** (Metrics and individual team can control) ·        **Engineering Management Level** (Metrics that require cross team collaboration or aggregate metrics across multiple teams) I also like to categorize the, based on what they will be used to solve: ·        **Progress**: used to solve visibility and decision making on the plan ·        **Bottlenecks and feedback** – as the name implies ·        **Quality** – Leading indicators connected to the Quality lagging indicator indirectly (Escaped Bugs in production) ·        **Skills** – track that we are slowly removing knowledge barriers inside teams or across teams ·        **Learning** – are we acting like a learning organization? The metrics are mostly quantitative (i.e they are numbers that can be measured), but a few are qualitative – in that you ask people how they feel or think about something. The ones I use are: ·        From 1-5 how much **confidence** do you have in the **tests** (that they will and can find bugs in the code if they arise)? (take the average of the team/cross teams) ·        **Same question but for the code** – that it does what I is supposed to do.  These are just surveys you can ask at each retrospective meeting, and they take 5 minutes to answer. ## Trend lines are your friends For all leading indicators and lagging indicators, you want to see **trend lines**, not just snapshots of numbers. Lines over time is how you see if you're getting better or worse. Don’t fall into the trap of a nice dashboard with large numbers on it, if your goal is **improvement over time**.  Trend lines tell  you if you're better this week than you were last week. Numbers do not. Numbers without context are not good or bad. Only in relation to last week/month/release etc. Remember, we're about change here. So trendlines are you friends. --- title: "Video: Lies, Damned Lies and Metrics" url: "https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics.md" canonical: "https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics" published: "2022-07-11T16:34:04.442000+00:00" updated: "2022-07-11T16:37:35.285000+00:00" tags: [] author: "Roy Osherove" description: "I spoke about what types of metrics and KPIs (including OKRs) you can use with continuous delivery, and which ones you might want to stop using, during my…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Video: Lies, Damned Lies and Metrics 11 July 2022 · 1 min read · Roy Osherove I spoke about what types of metrics and KPIs (including OKRs) you can use with continuous delivery, and which ones you might want to stop using, during my talk at GOTO 2019. Also discussed: Leading and Lagging Indicators. Enjoy. [![](https://pipelinedriven.org/img/yt/goihWvyqRow.jpg)](https://www.youtube.com/watch?v=goihWvyqRow) Lies, Damned Lies, and Metrics • Roy Osherove • GOTO 2019 · YouTube --- title: "If a Build Takes 4 hours, Run It Every 4 Hours" url: "https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours.md" canonical: "https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours" published: "2020-04-19T07:24:17.254000+00:00" updated: "2020-04-19T20:08:12.544000+00:00" tags: ["builds", "toc"] author: "Roy Osherove" description: "(Note: You might need to fix your environment bottlenecks first before being able to act on this blog post) Builds (i.e. compile + run tests + deploys +…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # If a Build Takes 4 hours, Run It Every 4 Hours 19 April 2020 · 4 min read · Roy Osherove _(Note: You might need to_ [_fix your environment bottlenecks first_](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments) _before being able to act on this blog post)_ Builds (i.e. compile + run tests + deploys + more tests etc…) represent a bottleneck for getting feedback on our code so we can be confident about making more changes or releases. There are two main factors here: - How often a build runs (Per commit, hourly, weekly etc..) - How long a build takes to finish Many companies are also in the process of reducing reliance on manual repetitive QA cycles, but still require some manual tests or verifications to be part of the process while the transition is happening (while building out more automated tests for example, or if there are some types of tests which currently don’t make sense automating). So we’ll add one extra factor here: - When does the manual verification take place Sometimes the manual verification can only take place after the build has been finished due to deployment requirements, environment requirements or a requirement for the built binaries to be available. This means the build acts as one of the **_constraints_** on our process. ### “How often?” Is the new Constraint We can assume the following based on [_Theory of Constraints_:](https://en.wikipedia.org/wiki/Theory_of_constraints) - The longer time passes between each build, the more changes are integrated in each build from various team members - This means the _batch size_ for changes is larger which means.. - we need to test/verify more things - manual testing takes longer, thus making the feedback loop longer - There are more compounded risks due to more changes which makes everyone more nervous, and more likely to ask for more verification to be do What can we do to reduce the batch size of each build, and get a faster feedback loop? The immediate answer is “run the build more often”. But, how often can that be? What I see many teams doing as an anti pattern, is doing the following thinking: > Because the build takes a long time (4 hours), let’s only run it at night or once a week But the thinking should be reversed: > Because the build takes 4 hours, let’s run it every 4 hours What do we get by that? - **The feedback loop time is reduced** to between 4-8 hours instead of 24 hours on a daily build, or a few days on a weekly build. Assuming a developer is making changes _right now_, they can get on the next “build train” that will start in a couple of hours and end 4 hours later. It’s not perfect but it’s much much better than waiting until the next day, or until the end of the week instead of waiting until 4pm to push that important change to the master branch. - **We always have a “latest stable build”** which is the last green build that happens every 4 hours. Assuming this also deploys to a dynamic demo environment, it means we always have a fresh-enough demo to show, and if not, just wait 4 hours. - **The batch size is much smaller** now: How many changes can fit into 4 hours? Less than 24 hours, that’s for sure. Which means **verification can be much faster too.** This also means that if I need to manually verify an issue I fixed, I don’t have to wait until tomorrow to verify it as part of an integrated build, I can wait 8 hours as a worst case scenario (same day half the time!). ### “How Long Does it Take?” is the next constraint Now, (and only now, after we’ve fixed a bigger constraint on when the build runs) that we reduced the time between builds, the build time is the next largest constraint that keeps the feedback loop from shortening. Every change we make to shorten the build time (by parallelizing tests or steps, for example) now has a compounded effect: **The build time is our feedback cycle time.** If the build now takes only one hour instead of four, we can run the build on an hourly basis. SO, manually verifying a change I made to an integrated build can now happen within 1-2 hours after making the change, committing and pushing it. Definitely same-day. What if the build takes 30 minutes? You get the idea. The loop is now 30-60 minutes long. ### Environments could be the original constraint ( Bottlenecks) This whole conversation might not be feasible for you because you have this issue. Sometimes teams are forced to run the build less because they don’t want to deploy to a static environment while someone else might be using it. I wrote about this before: [Static environments are a huge bottleneck](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments). So a huge factor here is your ability to have an environment per build that gets destroyed after a specified time or other constraint. The good news is that if you’re using docker and things like Kubernetes, this ability is 5 minutes away with K8s namespaces that are dynamically generated per build. 99% of static environments should be a thing of the past, and we need to know to ask for this from IT departments that keep the old ideas and just implement them on new tools that allow newer ideas. You might have to fix this issue, or at least create one static automation-only environment that humans do not touch, so you can open up this constraint and start working from the top of this article once this has been solved. Tagged: [builds](https://pipelinedriven.org/article/tag/builds), [toc](https://pipelinedriven.org/article/tag/toc) --- title: "Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)" url: "https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed.md" canonical: "https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed" published: "2019-12-06T22:10:04.805000+00:00" updated: "2019-12-06T22:10:07.299000+00:00" tags: [] author: "Roy Osherove" description: "“Co-Ops” is a term that one of the attendees at a talk I was giving about cooperative pipelines came up with. I wish I knew his name. I asked him to email…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed) 6 December 2019 · 1 min read · Roy Osherove “Co-Ops” is a term that one of the attendees at a talk I was giving about cooperative pipelines came up with. I wish I knew his name. I asked him to email me afterwards so I can credit him but he never did. Co-Ops describes the act of working with a cooperative pipeline: A Pipeline that every expertise contributes to: Dev, Test, Ops, Sec, Compliance etc.. a SINGLE Cooperative pipeline, with humans removed from tactical pipeline decision. I find it a really great naming scheme, and I think I’ll use it to title my book “Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes.” I’m still not fully “sold” on the full title. I fyou have ideas - I’d love to hear them. Your thoughts are welcome! --- title: "Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019)" url: "https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019.md" canonical: "https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019" published: "2019-11-20T22:44:53.528000+00:00" updated: "2019-11-20T22:47:23.885000+00:00" tags: [] author: "Roy Osherove" description: "Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019) 20 November 2019 · 1 min read · Roy Osherove [![](https://pipelinedriven.org/img/yt/bz4AKIIqGL4.jpg)](https://www.youtube.com/watch?v=bz4AKIIqGL4) Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube --- title: "Continuous Delivery Israel Meetup - First Meeting June 24 2019" url: "https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019.md" canonical: "https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019" published: "2019-06-02T08:16:32.780000+00:00" updated: "2019-06-02T08:16:33.002000+00:00" tags: [] author: "Roy Osherove" description: "I’m pleased to announce that we will have the first ever meeting Continuous Delivery Israel meetup, in the AWS Floor28 offices in Tel Aviv, June 24th, 2019,…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Continuous Delivery Israel Meetup - First Meeting June 24 2019 2 June 2019 · 1 min read · Roy Osherove I’m pleased to announce that we will have the first ever meeting **Continuous Delivery Israel** meetup, in the AWS Floor28 offices in Tel Aviv, June 24th, 2019, at 6PM. The Group’s page is located at CDIsrael.com [![CDIsrael.jpg](https://pipelinedriven.org/img/c/cdisrael-4a79cb.jpg)](https://pipelinedriven.org/img/c/cdisrael-4a79cb.jpg "View full size") If you would like to be a speaker or sponsor, [contact me.](https://pipelinedriven.org/contact) **AGENDA**: \------------- 18:00-18:30 Networking & Pizza 18:30 - 18:40 Goals for CD Israel + Logistics 18:40 - 19:20 Ant Weiss: Optimizing the Delivery Pipeline for Flow 19:20 - 19:30 Break 19:30 - 20:15 TBD - We are looking for one more speaker! **ABOUT**: Continuous Delivery is much more than simply Dev + Ops. It is also tech leads, compliance, security, project managers, scrum masters, architects, and everything in between - let's learn from each other! We deal with TECHNIQUES, PRACTICES, PROCESSES (and sometimes tools) that enable true continuous delivery - in large and small companies. We also discuss influencing such change in large and growing organizations. Bottlenecks & Pipelines: We feel feel pipelines are the core engines that can drive a software company, That Dev, Ops, Test, Sec, compliance should all be working together around common automated pipelines, and the processes, and metrics that push for and against adoption. Challenges and insights: We will learn from each other about challenges and techniques that enable true continuous delivery. Great for Ops, Dev, Security, compliance and agile leadership folks (PM, scrum masters, etc) --- title: "\"Pipeline Driven\" vs. \"We have Jenkins Jobs\"" url: "https://pipelinedriven.org/article/pipeline-driven-vs-we-have-jenkins-jobs.md" canonical: "https://pipelinedriven.org/article/pipeline-driven-vs-we-have-jenkins-jobs" published: "2019-05-03T00:02:56.276000+00:00" updated: "2019-05-03T00:02:57.420000+00:00" tags: ["philosophy"] author: "Roy Osherove" description: "High performing and continuously delivering organizations such as Netflix, Amazon, Google, Wix and others don’t just have pipelines. They don’t just…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # "Pipeline Driven" vs. "We have Jenkins Jobs" 3 May 2019 · 2 min read · Roy Osherove High performing and continuously delivering organizations such as Netflix, Amazon, Google, Wix and others don’t just _have_ pipelines. They don’t just _automate_ stuff and put jobs on it. That’s the easy part. Many organizations I have consulted for also have _pipelines_, jenkins jobs, teamcity projects, TFS configurations. But they don’t perform nearly as well fast or consistently (even though they are not happy with their current status) Places like Netflix and Amazon don’t just have _automated pipelines_. They are _driven_ by pipelines. What does that even mean compared to “traditional” pipeline approaches? It means important _decisions_ are made _in the pipelines_ without very little to no human interventions. These places _trust_ their pipelines to make decisions for them instead of putting humans in the bottleneck. In many places _humans_ take on the approving things like: - Merges between branches and pull requests - Deployment to environments - Setting up or killing environments - Enabling or disabling features - Pushing a feature to production (or other environments) - Security - Compliance - Configuration - Load balancing, scaling up or down - Tests passing or not - [Builds passing or not](https://pipelinedriven.org/article/red-builds-and-build-whisperers-trustworthy-noise-free-test-pipelines) - and more.. Companies can make all these decisions by humans and still be fully surrounded by a bunch of pipelines (automated jobs) that they trigger when they decide the time is right. Here lies the difference between a pipeline driven organization and a traditional one. To be _pipeline driven_ we remove the onus of approval from humans, and give it to pipelines. This requires several things: - That we _trust_ a pipeline enough to let it make good decisions - This means we need to _teach it_ to make good decisions by adding important steps to it in the form of various automated tests, security tests, compliance tests etc. - That we reframe questions from “did X say it’s OK” to “Did the pipeline pass?” - That the various knowledge silos in the organization shift their work towards mainly supporting and propping up pipelines relating to their knowledge that can make good decisions: - Ops folks will, among other things, help create infrastructure pipelines, infrastructure tests, environment configurations in code that are run in pipelines - Security folks will, among other things, help create security tests and specs that can be checked by automated pipelines and run by each team as needed. - Test folks will help developers learn how to write automated tests at various levels for the pipelines, and help with the test infrastructure - Developers will teach the pipeline (with the help of ops) how to deploy the application, which tests to run, and to do it per commit. - Everyone will write code that will run in a pipeline. - Everyone will slowly reduce their decisions and slowly let the pipeline make more and more of these decisions. And the end of the day, what separates the Netflix’s of the world from others is a simple truth: They worked really hard to create pipelines that they trust enough to make the hard decisions, and they use them continuously without waiting for humans in the process. Tagged: [philosophy](https://pipelinedriven.org/article/tag/philosophy) --- title: "Delivery Pipelines and Discovery Pipelines" url: "https://pipelinedriven.org/article/delivery-pipelines-and-discovery-pipelines.md" canonical: "https://pipelinedriven.org/article/delivery-pipelines-and-discovery-pipelines" published: "2019-02-20T14:23:08.702000+00:00" updated: "2019-02-20T14:23:12.735000+00:00" tags: [] author: "Roy Osherove" description: "In a previous post I discussed the notion of having two seperate pipelines. I want to revisit this, with a bit more details. When we’re talking about…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Delivery Pipelines and Discovery Pipelines 20 February 2019 · 3 min read · Roy Osherove In a [previous post](https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa) I discussed the notion of having two seperate pipelines. I want to revisit this, with a bit more details. When we’re talking about Continuous Delivery, we’re very interested in the speed of the feedback loop. When you’re expecting to merge into the master branch many times a day, a 5 minute detail in the run of a pipeline is crucial for making things flow productively and not waiting too much. People tend to create two types of pipelines to deal with this issue: **_Fast_ pipelines** and **_slow_ pipelines**. They’ll use the slow pipeline to run long running e2e tests for example. That really only slows down the rate of delivery. If those e2e tests are so important that you won’t delivery until you know their results, you’ll have to wait 24 hours to delivery, regardless of how small the changes are. ## Delivery Stoppers We can use the term **Delivery Stoppers** to create a logical bucket where we can put all things that we feel we cannot deliver unless they are “green”. e2e tests usually fit that bucket. So do unit tests, security tests, compilation steps etc. ## Discovery Pipelines What doesn’t fit that bucket? Code complexity is useful to know, but it usually doesn’t prevent us from delivering a new version when we know we need to fix some complicated code. Performance (under a specific threshold) is a useful metric, but should it break our build to know that our performance has increased by 1%? Or should it tell us that we need to add a backlog item to take care of that issue? If those things aren’t delivery stoppers, let’s call them **discoveries**. We want to know about them, but they should not be a build breaker. So now let’s consider our pipelines from that point: We can create **Delivery** **pipelines** - that run all things that can prevent delivery if they fail. - That end with a deployment of a deliverable product - That give us confidence we didn’t screw up badly from a functional stand point. - Has to run fast (even if it has slow tests - we’ll have to deal with making it run fast enough) We can create a **discovery pipeline** - that runs all the tasks that result in _interesting discoveries_ and KPI - Gives us new things to consider as technical debt or non functional requirements - Can result in new backlog items - Does not result in deployment. - Feeds a dashboard of KPIs - Can take a long time (that’s why it’s separate) Here’s an example of such pipelines: [![Delivery Pipeline and Discovery Pipeline](https://pipelinedriven.org/img/c/x-pipelines-testing-4f732c.jpg)](https://pipelinedriven.org/img/c/x-pipelines-testing-4f732c.jpg "View full size") Delivery Pipeline and Discovery Pipeline I know I said that every 5 minutes on a delivery pipeline makes a huge difference - so how come I’m willing to put long running 2e2 tests there? Because they are delivery stoppers. We have to have them running per commit. ## What do we do about the fact they are running slowly? - We can run the tests in parallel on multiple environments/build agents - We can split the tests into multiple tests suits and run them in even more parallelized processes on multiple environments. - We can optimize the tests themselves - We can remove unneeded/duplicated tests (if they already exist at a lower level such as unit test, API tests etc.) But we do not move them into the nightly pipeline. We deal with the fact that our delivery pipeline takes a long time, and don’t enable it to go further. Delivery pipelines and discovery pipelines - I think I can live with that. --- title: "Red Pipelines and Build Whisperers: Getting to a Trustworthy Noise Free Test Pipeline" url: "https://pipelinedriven.org/article/red-builds-and-build-whisperers-trustworthy-noise-free-test-pipelines.md" canonical: "https://pipelinedriven.org/article/red-builds-and-build-whisperers-trustworthy-noise-free-test-pipelines" published: "2019-02-18T12:48:40.887000+00:00" updated: "2019-06-18T22:13:33.768000+00:00" tags: [] author: "Roy Osherove" description: "People wonder why it takes such a long time to transform the way a large organization works. Here’s just a small nugget, a small tip of a small iceberg, in…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Red Pipelines and Build Whisperers: Getting to a Trustworthy Noise Free Test Pipeline 18 February 2019 · 4 min read · Roy Osherove People wonder why it takes such a long time to transform the way a large organization works. Here’s just a small nugget, a small tip of a small iceberg, in a small corner of a large building in a large state, filled IT folks trying to get their jobs done. And they have an nightly test pipeline. And nobody cares about it. I already explained [why this might be an anti pattern](https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa), but another thing that usually happens is that the nightly pipeline is not _trustworthy_: - There is a high _noise_ ratio. in many places, the “nightly” is almost always red and it takes a “specialist” (usually from the QA department) to “decrypt” the status of the build and tell developers if everything is actually OK or not. I like to call these pipelines: “**Red unless Green**” to signify that they are red by default, and green is almost by accident if it exists at all. - Because there is a high noise ratio - almost everyone disregards the results of the build, and so even if the build fails and finds real issues, they will be ignored and dismissed. - Many of the tests in the pipeline are fragile. They fail sometimes, and sometimes they don’t - but almost every build has some failing tests. Which leads to all the points above. When we have that kind of situation there are several things we might want to consider moving towards: 1. Getting the pipeline to become **trustworthy**: a.k.a _“Green unless Red”._ Right now it is at _“Red unless Green”_. 2. Getting the pipeline to run **fast** enough so that we can execute it on each commit instead of at night. 3. Merging the nightly test pipeline into the Delivery **per commit** pipeline. First, I want to make sure we understand how many risks are involved in “Red Unless Green” (**RUG**) pipelines: ## RUG (Red Unless Green) pipeline risks: 1. Tests may indicate a real issue but everyone dismisses the results because the build is always red. 2. Green tests are also ignored - so any confidence benefits we might get to help us run faster are dismissed. This often leads to manual testing since we cannot trust the nightly build. Wasting even more time. 3. Test cost goes unmaintained since nobody cares about the tests. 4. New tests are not added, or added as “lip service” since nobody cares about the feedback the build provides: both wasting time and providing no value. Let’s add _Build Whisperers_ to those risks. ## The Build Whisperer If you do have a person that is looking at the tests and can decipher their output to tell if they are actually in a good state, we call that person “Build Whisperer” - for no one else can understand the build they way that person does. That person is also human bottleneck and acting as a manual dashboard for the team. This presents several risks as well: - This person can take a while to decipher the results of the build before communicating status - The person can also make mistakes trying to understand the results - If the person is not available or working on other tasks - status is delayed, sometimes by days or even weeks. - If it is one person’s job to understand the build, then it’s nobody else’s job (“not my job” syndrome). This allows everyone else to dismiss responsibility for the build status. - If only one person watched the build, only one person cares about it. This might seem too much like the last point but let me put it this way: If many people watch the build, many people start slowly caring about it more and more. To get over thee risks - we decide, as a team, that the build will provide the final say on whether the product is deliverable or not. It then becomes our job to make the build robust enough and trustworthy enough that we want to listen to it and deliver based on its results, and not based on a human making a decision about manual test cases. ## What can we do? 1. Configure the nightly build to trigger as many times a day as possible (if it takes 5 hours to run, execute it 4 times a day). Don’t put it on the developer’s wall yet. This is just to increase the feedback loop speed as we fix the build. 2. DELETE or IGNORE all red tests. 3. Run the build if it is not running yet. 4. If there are red tests, repeat 1 -3. 5. If the build has been green for three days we have achieved a BASELINE GREEN BUILD. 6. At this point we can show the build on the Team’s Wall 7. At this point we can treat red test with actual work to fix the tests or the bugs they discover. We can finally feel worried about red tests, and gain more confidence if the tests are green. 8. We can slowly add back ignored tests that we deem necessary, adding them one by one, and fixing them in the process. 9. Continue adding new and old tests to the build incrementally to gain more confidence (not just E2E tests. [Use a test recipe](https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa).) 10. Make the build run faster (parallelize items, break up test suites, increase amount of agents and environments) so you can ultimately merge the nightly build into the per-commit delivery build. If point #2 scares you - remember that those tests are not really bringing to you value today. You are delivering your product even though they are failing. --- title: "A Pipeline Friendly Layered Testing Strategy & Recipe for DEV and QA" url: "https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa.md" canonical: "https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa" published: "2019-02-09T23:43:59.068000+00:00" updated: "2019-02-09T23:44:04.927000+00:00" tags: [] author: "Roy Osherove" description: "The key point of running a pipeline is to get feedback which in turn is supposed to provide us with Confidence. That’s because a pipeline is really just a…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # A Pipeline Friendly Layered Testing Strategy & Recipe for DEV and QA 9 February 2019 · 10 min read · Roy Osherove The key point of running a pipeline is to get feedback which in turn is supposed to provide us with Confidence. That’s because a pipeline is really just a big test - it’s either green or red; and it’s composed of multiple small tests that are either green or red. ## Types of feedback We can divide the types of tests in a pipeline into two main groups: ## Break/fail feedback - Provides a go-no-go for an increment of the code to be releasable and deployed - Great for unit tests, e2e, system tests, security tests and other binary forms of feedback ## Continuous Monitoring Feedback - An ongoing KPIs pipeline for non binary feedback - Great for code analysis and complexity scanning, high load performance testing, long running non functional tests that provide non binary feedback (“good to know”) To get faster feedback I usually have two pipelines, one for each type of feedback I care about: - Delivery pipeline: Pass/fail pipeline - Discovery Pipeline: Continuous KPI pipeline in parallel. ![Picture1.png](https://pipelinedriven.org/img/c/picture1-c5a1d3.png) The point of the Delivery feedback is to be used for continuous delivery – a go/no go that also deploys our code if all seems green. It ends with a deployed code, hopefully in production. The point of a Learning feedback is to provide learning missions for the team (Our code complexity is too high.. let’s learn how to deal with it), as well as show if those learnings are effective over time. It does not deploy anything except for the purpose of running specialized tests or analyzing code and its various KPIs. It ends with numbers on a dashboard. In this article I will focus on the confidence we gain from the delivery pipeline, and how we can gain even more speed out of it but still maintain a high level of confidence in the tests that it runs it’s important to note that speed is a big motivator for writing this article in the first place, and splitting into “discovery” and “delivery” pipelines is yet another technique to keep in your arsenal, regardless of what I write further down in this article. But let’s get back to the idea of “confidence”. ## What does confidence look like? In the delivery realm, what type of confidence are we looking for? - Confidence that we didn’t break our code - Confidence that our tests are good - Confidence that the product does the right thing ## Without Confidence… If we don’t have confidence in the tests this can manifest in two main ways: - If the test is red and we don’t have confidence in the test- you might hear the words “Don’t worry about it” - we go on as if everything is OK and assume the test is wrong. (potentially not paying attention to real issues in the code) - If the test is green and we don’t have confidence in the test - we still debug or do manual testing of the same scenario (wasting any time the test should have saved us). We might even be afraid to merge to the master branches, deploy or do other things that affect others. ## With Confidence… This confidence allows us to do wonderful things: - Deploy early and often - Add, change and fix code early and often. ## So what’s the problem? Let’s look at common types of tests that we can write and run in a delivery pipeline: ![Picture2.png](https://pipelinedriven.org/img/c/picture2-a41225.png) There are quite a few types to choose from. That means, there are usually several questions teams needs to decide when they come to the realization that they want to have lots of delivery confidence through automated tests: ## Which type of tests should we focus on? 1. How do we make sure the tests don’t take hours and hours to run so we can get faster feedback? 2. How can we avoid test duplication between various kinds of tests on the same functionality? 3. We are still (in some cases) working in a QA+Dev fashion. How can we increase collaboration and communication between the groups and create more knowledge sharing? 4. How can we inch towards DEV taking on automated test ownership as part of our transformation? 5. How can we be confident we have the right tests for the feature/user story? To start making an informed decision, let’s consider several key things for each test type: 1. How much confidence does the test provide? (higher up usually means more confidence – in fact, nothing beats an app running in production for knowing everything is OK) 2. How long is the test run time? (feedback loop time)(higher up means slower) 3. How much ROI to I get for each new test? (the first test of each type will usually provide the highest ROI. The second one usually has to repeat parts of the first so ROI is diminished) 4. How easy is it to write a new test? (higher up is usually more difficult) 5. How easy is it to maintain a test? (higher up is usually more difficult) 6. How easy is it to pinpoint where the problem is when the test fails? (lower down is easier) ## Where teams shoot themselves in the foot Many teams will try to avoid (rightfully so) repeating the same test twice, and since End to End (aka “e2e”) tests usually provide lots of confidence, the teams will focus mostly on that layer of testing for confidence. So the diagram might look like this for a specific feature/user story: ![Picture3.png](https://pipelinedriven.org/img/c/picture3-2bdb7c.png) It might also be the case that a separate team is working on the end to end tests while developers are also writing unit tests (yes, let’s be realistic – the real world is always shades of grey – and enterprises are slow moving giants. We have to be able to deal with current working practices during transforation), but there might be plenty of duplication or missing pieces between the two types of tests: ![Picture4.png](https://pipelinedriven.org/img/c/picture4-9f6eac.png) In both of these cases (and other variations in between) we end up with a growing issue: These e2e tests will take a long time to run as soon as we have more than a dozen. Each test can easily take anywhere from 30 seconds to a couple of minutes. It only takes 20-30 e2e tests to make the build run for an hour. That’s not a fast feedback loop. That’s a candidate for a “let’s just run that stuff at night and pray we see green in the morning” anti-pattern. Indeed, many teams at some point opt for that option and run these long running tests at night , possibly in a separate “nightly” pipeline which is different from the “continuous integration” pipeline. So now we might have the worst of both worlds: ![Picture5.png](https://pipelinedriven.org/img/c/picture5-d4a1c9.png) ## What’s wrong with this picture? - The pipeline that really determines the go-no-go delivery is the nightly pipeline. Which means 12-24 hours of feedback loop on the “real truth” about the status of the code (“come back tomorrow morning to see if your code broke anything”) . - Developers might get a false sense of confidence from just the CI (continuous integration) pipeline. - To know if you can deliver, you now have to look at two different locations (and it might also mean different people or departments are looking at each board instead of the whole team) ## The best of both worlds? Can we get high confidence with fast feedback? Close. What if we try to get the confidence of e2e tests but still get some of the nice fast feedback that unit tests give us? Let’s try this scheme on for size: - Given Feature 1 - We can test the standard scenario 1.1 as an e2e test - For any variation on that scenario, we always write that variation in either a unit test or an integration test (faster feedback). Now we get both the “top to bottom” confidence and the added confidence that more complicated variations in the logic are also tested in fast tests at a lower level. Here’s how that might end up looking: ![Picture6.png](https://pipelinedriven.org/img/c/picture6-2ce904.png) ## Creating a Test Recipe Based on this strategy, here’s a simple tactical maneuver that I’ve found helpful in various teams I’ve consulted with: Before starting to code a feature or a user story, the developer sits with another person to create a “Test Recipe”. That other person could be another developer, a QA person assigned ot the feature (if that’s your working process), an architect, or anyone else you’d feel comfortable discussing testing ideas with. A Test Recipe as I like to call it is simply a list of the various tests that we think might make sense for this particular feature or user story, including at which level each test scenario will be located in. ## A test recipe is NOT: - A list of manual tests - A binding commitment - A list of test cases in a test planning piece of software - A public report, user story or any other kind of promise to a stakeholder. - A complete and exhaustive list of all test permutations and possibilities At its core it’s a simple list of 5-20 lines of text, detailing simple scenarios to be tested in an automated fashion and at what level. The list can be changed, added to or detracted from. Consider it a “comment”. I usually like to just put it right there as a bottom comment in the user story or feature in JIRA or whatever program you’re using. Here’s what it might look like: ![Picture7.png](https://pipelinedriven.org/img/c/picture7-78df15.png) ## When do I write and use a test recipe? Just before coding a feature or a user story, sit down with another person and try to come up with various scenarios to be tested, and discuss at which level that scenario would be better off tested. This meeting is usually no longer than 5-15 minutes, and after it, coding begins, including the writing of the tests (if you’re doing TDD, you’d start with a test first). In orgs where there are automation or QA roles, the developer will take on writing the lower level tests and the automation expert (Yes QA without automation abilities wil need to learn them slowly) will focus on writing the higher-level tests in parallel, while coding of the feature is taking place. If you are working with feature toggles, then those feature toggles will also be checked as part of the tests, so that if a feature if off, it’s tests will not run. ## Simple rules for a test recipe 1. **Faster**. Prefer writing tests at lower levels - Unless a high level test is the only way for you to gain confidence that the feature works - Unless there is no way to write a low level test for this scenario that makes you feel comfortable 2. **Confidence**. The recipe is done when you can tell yourself “If all these tests passed, I’m feel pretty good about this feature working.” If you can’t say that, write more scenarios that would make you say that. 3. **Revise**: Feel free to add or remove tests from the list as you code, just make sure to notify the other person you sat with. 4. **Just in time**: Write this recipe just before starting to code, when you know who is going to code it, and coding is about to start the same day. 5. **Pair**. Don’t write it alone if you can help it. Two people think in different ways and it’s important to talk through the scenarios and learn from each other about testing ideas and mindset. 6. **Don’t repeat yourself.** If this scenario is already covered by an existing tests (perhaps from a previous feature), there is no need to repeat this scenario at that level (usually e2e). 7. **Don’t repeat yourself again**. Try not to repeat the same scenario at multiple levels. If you’re checking a successful login at the e2e level, lower level tests will only check variations of that scenario (logging in with different providers, unsuccessful login results etc..). 8. **More, faster.** A good rule of thumb is that to end up with a ratio of at least 1 to five between each level (for one e2e test you might end up with 5 or more lower level other scenario tests) 9. **Pragmatic**. Don’t feel the need to write at test at all levels. For some features or user stories you might only have unit tests. For others you might only have API or e2e tests. As long as you don’t repeat scenarios, and as long as you ## Benefits of Layered Test Strategies and Test Recipes Using this strategy, we can gain several things we might not have considered: ## Faster feedback OK, that one we did consider, but it’s the most notable. ## Single delivery pipeline If we can get the tests to run fast, we can stop running e2e tests at night and start putting them as part of the CI pipeline that runs on each commit. That’s real feedback for developers when they need it. ## Less test duplication We don’t waste time and effort writing and running the same test in multiple layers, or maintaining it multiple times if it breaks. ## Knowledge sharing Because test recipes are done in pairs, there is much better knowledge sharing and caring about the tests, especially if you have a separate QA department. Test recipes will “force” devs and QA to have real conversations about mindset and testing scenarios and come up with a better, smaller, faster plan to gain confidence, together. Real teamwork is closer. ## Thoughts? Hope you find this useful. Feel free to drop me a line on twitter @royosherove or email me \[email removed\] (or comment on this post!) ### PS Test strategies aren’t usually enough when you already have a large=ish list of e2e long running tests. You’ll still have to optimize the pipelines. There are many ways, but a rather simple one (given enough build agents) is to parallelize the steps in the the pipelines. Here’s a simple example of that: ![Picture8.png](https://pipelinedriven.org/img/c/picture8-0e082e.png) --- title: "Always start with a \"Why\" when transforming to DevOps." url: "https://pipelinedriven.org/article/always-start-with-a-why-when-transforming-to-devops.md" canonical: "https://pipelinedriven.org/article/always-start-with-a-why-when-transforming-to-devops" published: "2018-10-21T18:08:30.427000+00:00" updated: "2018-10-21T18:08:31.384000+00:00" tags: [] author: "Roy Osherove" description: "I’ve been to quite a few DevOps and Agile transformations, big and small. Some just want to up the coverage of their tests (not recommended!), others want…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Always start with a "Why" when transforming to DevOps. 21 October 2018 · 3 min read · Roy Osherove I’ve been to quite a few DevOps and Agile transformations, big and small. Some just want to up the coverage of their tests (not recommended!), others want to revamp their team structures, and some want to change their process. There are other points of view as well: Many internal and external consultants who want to make a difference try changing things but end up failing for many reasons. My advice, which I’ve learned the hard way (like anything of true value) - is to have a good “Why” in your head at all times. “Why am I doing this?”, “What’s the point of this change?”, “What are we trying to fix by changing this?” , “What is the cost of NOT changing this?”. You have to have a really good answer for all of these, which are just a variation of the age old “What’s your motive?” question. Not only do you need to have a good answer, you need to fully believe that this answer is what would really drive things for the better. If you’re aiming to change something and you yourself are not sure there’s any point in doing this, or tht it could actually succeed, if you can’t even imagine a world where that success has been accomplished, a realizty where things have changed (be it a month from now or 5 years from now) - it’s going to be very difficult. Change always has detractors and naysayers. And even though from their point of view there might be truth to their reasons for not agreeing with you, their truth should not hinder your own belief that this change is needed. Where this “Why” comes in handy the most, for me, is when someone comes with a really good argument for “Why not”. You can’t know everything and you can bet on being surprised at some of the good reasons people give why things shouldn’t be done or cannot. Or why they think things will not succeed, or are doomed to make things worse. It’s really handy because at those dark times, where you’re even unsure in yourself that this is the right thing to do, you can come back to your original “Why”. And remind yourself why this makes sense to you. And you can use that “Why” to measure any fact you encounter, where it still holds true or not. If it no longer holds true - you might have truly discovered or learned something new and would need need to possibly change strategy or goals. Here’s an example of “Why” that I have used: “Developers should have fast feedback loops that enable them to move fast with confidence”. Strategy: - Teach about single trunk branching technique - Teach about Feature Flags - Help team design a new branching model that reduces amount of merges - Help team design pipeline that triggers on each commit - Get the pipeline to run fast, in parallel, on multiple environments - Teach team about unit testing, TDD and integration tests That shoudl get a team far enough on that “Why”. And here’s what happens in real life, especially in large organizations: Every bullet point above will be challenges, cross examined, denounced as the devil, offered multiple variations that seem to be close but are not, and very few people will be able to automatically support it in many teams. But this “Why” is what keeps my moral compass afloat. No matter what is the discussion - and especially if it becomes confusing, or goes deep into alternative suggestion land - I keep asking myself if that “Why” is being answered. And I can use it as my main reason for each one of these things, because it is absolutely true in my mind - these points help push that “Why” - and doing them in other ways might seem like it’s close but can push us farther away from it. Always have a “Why” - and you’ll never be truly stumped, even when confusion takes over. You could have many “Whys” - and not all of them will be in play at all times. But for each action you push for, there should be a “Why” behind it. In my mind it reminds me a bit of TDD - any production code has a reason to exist: a failing test at some point that pointed out that this functionality is missing. Tests are the “Why” of code functionality. In DevOps and Continuous Delivery, a big “Why” can be “We want to [reduce our cycle time](https://pipelinedriven.org/article/ten-ideas-for-things-you-can-measure-as-a-team-on-your-devops-journey). “ Then from that you can derive smaller “Whys”, but they all live up to the big one, and support it. --- title: "Ten Devops & Agility Metrics to Check at the Team Level" url: "https://pipelinedriven.org/article/ten-ideas-for-things-you-can-measure-as-a-team-on-your-devops-journey.md" canonical: "https://pipelinedriven.org/article/ten-ideas-for-things-you-can-measure-as-a-team-on-your-devops-journey" published: "2018-03-10T13:08:23.186000+00:00" updated: "2018-03-10T13:09:00.901000+00:00" tags: [] author: "Roy Osherove" description: "When I coach teams that are getting into the DevOps and Continuous Delivery mindset, a common question that comes up is \"What should we measure?\" Measuring…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Ten Devops & Agility Metrics to Check at the Team Level 10 March 2018 · 4 min read · Roy Osherove When I coach teams that are getting into the DevOps and Continuous Delivery mindset, a common question that comes up is "What should we measure?" Measuring is a core piece of change - how do you know you're progressing without measuring anything? Here are ten ideas for things you can measure to see if your team is getting closer to a DevOps and continuous delivery skillset. It's important to realize that what we are measuring are end symptoms - results. The core behaviors that need to change can be varied quite a bit, but at the end of the day, we want to see real progress in things that matter to us from a continuous delivery perspective. 1. **Cycle time.** (you want to see this number going down)  If you put a GoPro on a user story, from the moment it enters the mind of a customer or PO, and track what it goes through, to the point of being active in production, you get a calendar-time number that represents your core delivery cycle time. It could take weeks, months and sometimes years in large organizations. It usually id a big surprise. I'll write about this more in a separate blog post.  The idea is to see cycle time reduced over time, so you actually deliver faster and be more competitive. 2. **Time from red build to green build** (you want to see this number going down) - Take the last instances of a red-to-green build (count from the first red build, until the first green build after that) to get how long on average it takes to make a red build green. This is how effective your team is with dealing with a build failure. Build failures a re a good thing - they tell us what's really going on. We should not avoid them. But we should be taking care of them quickly and efficiently (for example you can set up "build keeper" shifts -every day someone else is in charge of build investigations and pushing the issue to the right people in the team. 3. **Amount of open pull requests on a daily basis, closed pull requests, coupled with the avg. time a pull request has been open.** (you want to see closed requests going up, request time going down and open requests being stable or going down).  This gives us a measure of team communication and collaboration - how often does code get reviewed, and how often is code code stuck waiting for a review. A trend of open pull requests going up could mean the team has a bottleneck in the code review area. The same is true for very long pull request times. 4. **Frequency of merges to Trunk**. (this should be going up or staying stable) If your code gets merged to trunk every few days or weeks, it means that whatever it is your build pipeline is building and delivering is days old or weeks old code. It also is a path to many types of risks such as: not getting feedback fast enough on how your code integrates with everyone else, your code not being deployed and available to turn on with a feature flag, and generally it's a pathway for people who are afraid of exposing their work to the world, thus potentially creating hours and sometimes days of pain down the line.  5. **Test Code Coverage** (coupled with test reviews) (you want to see this go up or stay stable at a high level, while watching closely for quality of code reviews). I always like to say that low code coverage means only one thing - you are missing tests. but high code coverage is meaningless unless you have code reviewed, because human nature leads us to fulfill whatever we are measured on. so sometimes you can see teams writing tests with no asserts just to get high code coverage. this is where the code reviews come in.    6. **Amount of tests** (this should obviously be going up as you add new functionality to your product). 7. **pipeline run time** . (this should be declining or staying at a low level). The slower your automated build pipeline is the slower your feedback is.  This helps you know if the steps you are taking also help increase the feedback cycle. 8. **pipeline visibility** in team rooms (you want to see this go up or stay stable at a high level). This is a metric that tells you about commitment to visual indicators, information radiators etc. It's a small but important part of team non verbal communication and increases the team's ability to respond quickly to important events.  9. **team pairing time** (should be going up or stay stable at a medium or high level) - we can measure this to see if we have knowledge sharing going on. 10. **amount of feature flags** -(should be going up as team learns about feature flags, and then stay stable. if it continues to increase it means you're not getting rid of feature flags fast enough which can lead to trouble down the line.  Two bonus metrics: 1. **feature size estimate** (should be staying stable or going down) - helps to track how well the team estimates feature sizes or to check the variance of the feature sizes you estimate. 2. Bus factor count - (should be going down and staying down) how many people [are bus factors](http://5whys.com/blog/the-bus-factor-why-your-best-developer-is-your-biggest-probl.html)? --- title: "Ephemeral Environments for DevOps: Why, What and How?" url: "https://pipelinedriven.org/article/ephemeral-environment-why-what-how-and-where.md" canonical: "https://pipelinedriven.org/article/ephemeral-environment-why-what-how-and-where" published: "2017-03-30T17:31:29.107000+00:00" updated: "2019-03-02T19:11:50.283000+00:00" tags: [] author: "Roy Osherove" description: "In the previous post I discussed the issues with having static environments . In this post I cover one of the solutions for those issues: Ephemeral…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Ephemeral Environments for DevOps: Why, What and How? 30 March 2017 · 4 min read · Roy Osherove In [the previous post I discussed the issues with having static environments](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments). In this post I cover one of the solutions for those issues: Ephemeral environments. Ephemeral environments are also sometimes called “Dynamic environments”, “Temporary environments”, “on-demand environments” or “short-lived environments”. The idea is that instead of environments “hanging around” waiting for someone to use them, the CI/CD pipeline stages are responsible for instantiating and destroying the environments they will run against. For example, we might have a pipeline with the following stages: 1. Build 2. Test:Integration&Quality 3. Test:functional 4. Test:Load&Security 5. Approval 6. Deploy:Prod [![(Image courtesy of cloudbees)](https://pipelinedriven.org/img/c/pipeline-visualization-f97f5a.png)](https://www.cloudbees.com/blog/top-10-best-practices-jenkins-pipeline-plugin) (Image courtesy of cloudbees ) **In a traditional static environment configuration**, each stage (perhaps except the build stage) would be configured to run again a static environment that is already built and is waiting for it, or some steps might share the same environment, [which causes all the issues I mentioned previously](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments). **In an ephemeral environment configuration**, each relevant stage would contain two extra actions: one at the beginning, and one at the end, that spin up an environment for the purpose of testing, and spin it down at the end of the stage.  The first step(1) is to compile and run fast unit tests, followed by putting the binaries in s aspecial binary repository such as artifactory. There is also an stage (2) that creates a pre-baked environment as a set of AMIs of VM images (or containers) to later be instantiated. : 1. Build & Unit Test 1. Build Binaries, run unit tests 2. Save binaries to artifact managemnt 2. Pre-Bake Staging Environment 1. Instantiate Base AMIs 2. Provision OS/Middleware components 3. Provision/Install application 4. Save AMIs for later instantiation as STAGING environment (in places such as S3, artifactory etc.) 3. Test:Integration&Quality 1. Spin up staging environment 2. Run tests 3. Spin down staging environment 4. Test:functional 1. Spin up staging environment 2. Run tests 3. Spin down staging environment 5. Test:Load&Security 1. Spin up staging environment 2. Run tests 3. Spin down staging environment 6. Approval 1. Spin up staging environment 2. Run approval tests/wait for approval and provide a link to the environment for humans to look into the environment 3. Spin down staging environment 7. Deploy:Prod 1. Spin up staging environment 2. Data replication 3. Switch DNS from old production to new environment 4. Spin down old prod environment (this is a very simplistic solution) ## A few notes: ### ·       **Pre-Baked Environment:** Notice that the environment we are spinning up and spinning down is always the same environment, and it is a staging environment with the application pre-loaded on top of it. Staging environments are designed to look exactly like production (in fact, in this case, we are using staging as a production environment in the final stage). The reason we are always using the same environment template, is because: - This provides **environmental consistency** between all tests and stages, and removes any false positives or negatives. If something works or doesn’t work, it is likely to have the same effect in production. - Environments are pre-installed with the application, which means that we are always testing the exact same artifacts, so we get **artifact consistency** - Because environments are pre-installed, we are also in-explicitly testing the installation/deployment scripts of the application. ###  **Only one Install.** Also notice that there is no “installation”  after the pre-baking stage. – which means we also don’t “deploy” into production. We simply “instantiate a new production in parallel”. We “install-once, promote many” which means we get **Installation consistency** across the stages. ### **Blue-Green Deployment** Deploying to production just means we instantiate a new pre baked environment in the production zone (for example a special VPC if we are dealing in AWS) which would run in parallel with the “real” production. Then we slowly soak up production data, let the two systems run in parallel, and eventually either switch a DNS to the new server, or slowly drain the production load balancer into the new server (there are other approaches to this that are beyond the scope of this article. ## Speed Another advantage of this set up is that because each stage can have its own environment, we can run some stages in parallel, so, in this case we can run all the various tests in parallel, which will save us valuable time: 1. Build & Unit Test 1. Build Binaries, run unit tests 2. Save binaries to artifact managemnt 2. Pre-Bake Staging Environment 1. Instantiate Base AMIs 2. Provision OS/Middleware components 3. Provision/Install application 4. Save AMIs for later instantiation as STAGING environment (in places such as S3, artifactory etc.) 3. **HAPPENING IN PARALLEL:** 1. **_Test:Integration&Quality_** 1. _Spin up staging environment_ 2. _Run tests_ 3. _Spin down staging environment_ 2. **_Test:functional_** 1. _Spin up staging environment_ 2. _Run tests_ 3. _Spin down staging environment_ 3. **_Test:Load&Security_** 1. _Spin up staging environment_ 2. _Run tests_ 3. _Spin down staging environment_ 4. Approval 1. Spin up staging environment 2. Run approval tests/wait for approval and provide a link to the environment for humans to look into the environment 3. Spin down staging environment 5. Deploy:Prod 1. Spin up staging environment 2. Data replication 3. Switch DNS from old production to new environment 4. Spin down old prod environment (this is a very simplistic solution) ## How? One tool to look into for managing environments and also killing them easily later would be Chef:Provision, which can be invoked from the jenkins command line, but also saves the state later for spinning down an environment. It also follows the [toolchain values we discussed before on this blog.](https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains) In the docker world, given a pre-baked set of docker images, we can use kubernetes to create ephemeral environments very easily, and destroy them at will. [Jenkins-X would be a good tool to look into](https://jenkins-x.io/) specifically for those types of environments. --- title: "The four biggest issues with having static environments" url: "https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments.md" canonical: "https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments" published: "2017-03-30T17:12:52.430000+00:00" updated: "2017-03-30T17:19:57.134000+00:00" tags: [] author: "Roy Osherove" description: "An environment is a set of one or more servers, configured to host the application we are developing, and with the application already installed on them,…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The four biggest issues with having static environments 30 March 2017 · 2 min read · Roy Osherove An **environment** is a set of one or more servers, configured to host the application we are developing, and with the application already installed on them, available for either manual or automated testing. ## What are static environments? Static environments are environments that are long lived. We do not destroy the environment, but instead keep loading it with the latest and greatest version of the application. For example, we might have the following static environments: “DEV”, “STAGE” and “PROD”. Each one is used for a different purpose and by different crowds. What’s common about them is that they are all **long lived** (sometimes for months or years), and this creates several issues for the organization: 1.     **Environment Rot**: As time goes by, the application is continuously installed on the environments and configuration is done on them (manually). This creates an ongoing flux of changes to each environment that leads to several problems a.     **Inconsistency** between environments (false positives or negatives) - Any **deployment or tests results** you get in one environment may not reflect what you actually get in production. For example, tests that pass in “Model” could be passing due to a specific configuration in MODEL that does not exist in other environments, meaning we’d get a _false positive_. - **Bugs** that happen in one of the environments might not happen in production, which is a _false negative_. b.     **Inability to reproduce issues** between environments If a bug is manifested in one environment but cannot be reproduced in another, due to the fact that the environments are different “pets” **2.**    **Long and costly maintenance times** Because environments are treated as “pets” (i.e you name them, you treat them when they are sick, each one is a unique snowflake that has its ups and downs), it takes a lot of time and manual, error prone activities, to maintain the environment and bring it up if it crashes. This causes a delay whenever a team needs to test the product on an environment. **3.**    **Queuing** Because of teams waiting to deploy to an environment, and because there is a limited number of these environments (because they are costly to set up, maintain and pay for to keep running 24-7), queues start to form as teams await environments to become available. This queueing can also be caused because multiple teams are expecting to use the same environment, and so each team waits for other teams to finish working with the environment before they can start working. This causes release delays. **4.**    **Waste of money**  Static environments usually run 24-7, and in a private or public cloud scenario this might mean paying per hour per machine or VM instance. However, most environments are only used during work hours, which means in many organizations **up to 16 hours of idle paid time.** ## **Solution:**  In the next post I'll cover ephemeral environments and how they solve the issues mentioned here. --- title: "Continuous Delivery Values" url: "https://pipelinedriven.org/article/continuous-delivery-values.md" canonical: "https://pipelinedriven.org/article/continuous-delivery-values" published: "2017-03-10T21:33:29.052000+00:00" updated: "2017-03-10T21:44:55.963000+00:00" tags: [] author: "Roy Osherove" description: "I mentioned in my last post about the toolchain needing to respect the continuous delivery values . What are those values? Many of them derive from lean…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Continuous Delivery Values 10 March 2017 · 1 min read · Roy Osherove I mentioned [in my last post about the toolchain needing to respect the continuous delivery values](https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains). What are those values? Many of them derive from lean thinking, eXtreme Programming ideas and the book "[Continuous Delivery](http://amzn.to/2nob369)". Michael Wagner, a colleague and mentor of mine at Dell EMC,has described them as: The core value is :  ## Our highest priority is to satisfy the customer through early and continuous delivery of valuable software ## The principles are: - The process for releasing/deploying software MUST be repeatable and reliable - Automate Everything down to bare metal - If something’s difficult or painful, do it more often - Keep EVERYTHING in source control - Done means “released” - Build quality in - Everyone has responsibility for the release process - Improve Continuously ## The Practices are: - Build binaries (or image, or containers, or any artifact that will be deployed) only once, promote many - Use precisely the same mechanism to deploy to every environment - Smoke test your deployment - Organizational policy is implemented as automated steps in the pipeline - If anything fails, stop the line --- title: "Guidelines for selecting tools for continuous delivery toolchains" url: "https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains.md" canonical: "https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains" published: "2017-03-10T17:19:48.617000+00:00" updated: "2017-03-21T20:37:30.878000+00:00" tags: [] author: "Roy Osherove" description: "If the tool you use does not support continuous delivery values , you're going to have a bad time implementing CI/CD with fully automated pipelines. Here…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Guidelines for selecting tools for continuous delivery toolchains 10 March 2017 · 1 min read · Roy Osherove If the tool you use does not support [continuous delivery values](https://pipelinedriven.org/article/continuous-delivery-values), you're going to have a bad time implementing CI/CD with fully automated pipelines. Here are some rules for the road: 1. [The first rule is: Don't select your toolchain until you have designed the pipeline you want to have](https://pipelinedriven.org/article/the-first-rule-of-continuous-delivery-toochains) 2. Every action or configuration can become **code in source control** so you can version things and get an audit trail on changes 3. Everything that can be invoked or configured has an **automation end point** through _command line_ or _APIs_ 4. Every command line can return **zero for success or non zero for failures** 5. If you have to **log in to a UI** to be able to configure or execute something, you're going down the wrong path -  Your CI/CD tools (like Jenkins) is your user, not humans. 6. **Queues should be avoidable**: if you can only do one task at a time, but you have multiple builds that need to use the tool, you'll have a queue. The tool should be able to support parallel work by multiple pipelines or any other support for avoiding such queuing - as long as it enables pipelines to run faster, not slower, but still give all the feedback you need into the pipeline. 7. **Results should be easily visible in the pipeline, or importable** **via API or command line:** You need to be able to see the results easily in the pipeline log so that you can easily understand pipeline issues without going through different teams and tools. --- title: "The first rule of continuous delivery toolchains" url: "https://pipelinedriven.org/article/the-first-rule-of-continuous-delivery-toochains.md" canonical: "https://pipelinedriven.org/article/the-first-rule-of-continuous-delivery-toochains" published: "2017-03-10T17:06:29.643000+00:00" updated: "2017-03-30T17:35:26.408000+00:00" tags: [] author: "Roy Osherove" description: "Continuous delivery transformations are hard enough in many areas already: changing people's behavior, changing policies, architecting things differently -…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The first rule of continuous delivery toolchains 10 March 2017 · 1 min read · Roy Osherove Continuous delivery transformations are hard enough in many areas already: changing people's behavior, changing policies, architecting things differently - all these things are difficult on their own.   Having a toolchain that prohibits you from achieving your pipelines in a way that fully supports CI/CD is something you can try to avoid off the bat.  The simple rule is: ## Don't select your toolchain before you've designed the pipeline you wish to have Don't decide you're going to use a specific tool for the new way of working before you've done at least the following: - Choose a project in your organization that exemplifies an "average" project (if there are many types choose two or three). - Do a [value stream mapping](https://en.wikipedia.org/wiki/Value_stream_mapping#In_software_engineering) of that project from conception to cash  - from the point of an idea, to that getting into a backlog, to that code ending up in production. Then Time the value stream and see how long each step takes. - Once you have a value stream, design a conceptual pipeline that tries to automate as much of the value stream as possible (some things will not be able to be automated without the change of organizational policies) Now that you have  conceptual pipeline (or several), you can start deciding which tools fit best for the actions you would like to pull off in the pipeline. What you'll find is that some tools will not lend themselves easily into the pipeline work, and some tools will fit in just right. ## Remember: the pipeline drives the tool selection. Not the other way around. You should not let the tool dictate how your pipeline works because you're trying to change the way the organization works, not map the organization to a tool that might hider your progress. --- title: "The four most common root causes that slow down enterprise continuous delivery" url: "https://pipelinedriven.org/article/the-four-most-common-root-causes-that-slow-down-enterprise-continuous-delivery.md" canonical: "https://pipelinedriven.org/article/the-four-most-common-root-causes-that-slow-down-enterprise-continuous-delivery" published: "2017-03-09T06:19:23.777000+00:00" updated: "2017-03-09T07:34:20.015000+00:00" tags: [] author: "Roy Osherove" description: "In my journey as a developer, architect, team leader, CTO, director, coach and consultant for software development, the two most common \"anti patterns\" I…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The four most common root causes that slow down enterprise continuous delivery 9 March 2017 · 8 min read · Roy Osherove In my journey as a developer, architect, team leader, CTO, director, coach and consultant for software development, the two most common "anti patterns" I come across int he wild that generate the most problems are: 1. Manual Testing 2. Static Environments 3. Manual configuration of those environments 4. Organizational Policies ## Manual Testing This one seems pretty straight forward: Because testing is manual, it takes a really long time. Which slows down the rate of releases as everyone waits for testing to be done. There are other side effects to this approach though: ### Because testing is manual: - It is hard to **reproduce bugs**, or even **the same testing steps** continuously - It is prone to **human error** - It is **boring** and creates **frustration** to the people who do it (which leads to the common feeling among people that 'QA is a stepping stone to becoming a developer' - and we do not want that! We want testers who love their job and provide great value!) - The list of **manual** tests **grows exponentially** - and so does the manual work involved. as more features are added, you get very low code coverage, since people can only spend so much time testing the release before the company goes belly up. - It creates a **knowledge silo** in the organization, a "throw it over the fence" mentality for developers who are only interested in hitting a date, but not wether the feature quality is up to par ('not my job").  - This knowledge silo also creates psychological "bubbles" around people in different groups, that 'protect' them from information of what happens before and after they do their work. Essentially meaning that **people stop caring or have little knowledge of how they contribute and where they stand on the long pipeline of delivering software to production** ("Once I sign off on it, I have no idea what is the next step - I just check a box in "ServiceNow" and move on with my life" ) . - **You never have enough time to automate your tests, because you are too busy testing things manually!** So manual testing is both a scaling issue, a consistency issue and a cause for people leaving the company. ## Static Environments An environment is usually a collection of one or more servers sitting inside or outside the organization (private or public cloud, and sometimes just some physical machines the developers might have lying around if they get really desperate). In the traditional software development life cycle that uses pipelines, code is built, and then promoted through the environments, that progressively look more and more like production, with "Staging" usually being the final environment that stands between the code and production deployment. How it moves through the environments changes: some organizations will force to merge to a special branch for each type of environment (i.e "branch per promotion"). This is usually not recommended, because it means that for each environment the code will have to be built and compiled again, which breaks a cardinal rule of CI/CD: Build only once, promote many - that's how you get consistency. But that's a subject for another blog post. More problematic is the fact that the environments are "static" in the first place. Static means they they are long-lived: an environment is (hopefully) instantiated as a set of virtual machines somewhere in a cloud, and then is declared "DEV" or "TEST" or "STAGE" or any other name that signified where it is supposed to be used in the organizational pipeline that leads to production. Then a specific group of people tend to get access to it and use it to (as we said in the first item) test things manually. But even if they test things with automated tests, ### The fact that the environment is long lived is an issue for multiple reasons: - Because they are static, there is only a fixed number of such environments (usually a very low number), which can easily cause **bottlenecks** in the organization: "We'd like to run our tests in this environment, but between 9-5 some people are using it"  or "We'd like to deploy to this environment, but people are expecting this environment to have an oder version"     - Because they are static, they become "stale" and "dirty" over time:  Each deploy, or configuration change becomes a patch on top of the existing environment leading it to become a **unique "snowflake"** that cannot be recreated elsewhere. This creates **inconsistency between the environments**, which leads to problems like "**The bug appears in this environment, but not in that environment, and we have no idea why"** - **It costs a lot of money:** An environment other than production, usually is only utilized in times when people are at work, otherwise it just sits there, ticking away CPU time on amazon , or electricity on prem, and is a waste of money.  Imagine that you have a fleet of 100 testing machines in an environment that are utilized in parallel during load testing, but that only happens once a day, or three times a day, but 12 hours a day these 100 machines just sit there ,costing money.  So static environments are both a scale issue and a consistency issue. ## Manual Configuration of environments In many organizations the static environments are manually maintained by a team of people dedicated to the well being of those snowflakes: they patch them, they mend their wounds, they open firewall ports, they provide access to folders and DNS and they know those systems well.  But  ### Manual configuration is an issue for multiple reasons: - Just like manual testing, it **takes a lot of time to do anything useful** with an environment : bringing it up and operationalizing it takes a long time and painstaking dedication over multiple days sometimes weeks of getting everything sorted with the internal organization. Simple questions rise up and have to be dealt with: "who pays for this machine? Who gets access? what service accounts are needed? how exposed is it? Will it contain information that might need encryption? Will compliance folks have an issue with this machine being exposed to the public internet?  - these and more have to be answered, and usually be multiple groups of people within a large organization.   The same with any special changes to an existing environment, or, god forbid, debugging  stray application that's not working on this snowflake, but does work everywhere else - a true nightmare to get such access if you've ever been involved in such an effort.     - Not only is it slow, but it also **does not scale**:  Onboarding new projects in the organization will take a long time - if they need a build server, or a few "static" environments: that's days or weeks away. So this slows the rate of innovation.  - There is **no consistency** between the environments, since all things are done manually, and people often find it hard to repeat things exactly. No consistency: the feedback you get from deploying onto that environment might not be true, and thus the time to really know if your feature actually does work might be very long: maybe you'll find out only in production, a few weeks or months from now. and we all know that the longer you wait to get feedback, the more it costs to fix issues. - Importantly: t**here is no record of any changes made** to environment configuration: all changes are manual so there is no telling what changes were made to an environment, as well as rolling back changes is difficult or impossible.  So manual configuration of environments hurts time to market, consistency, operational cost and the rate of innovation. ## Organizational Policies If your organization requires manual human intervention to approve things that go into production , for _everything then this policy can cause several issues:_ - No matter how much automation you will have, the policy that drives the process will force manual intervention and automation will not be accepted (especially auto-deploy to production) - It will slow down time to market - It will make small changes take along time, and large changes take even longer (if not the same) ## Here is the guidance I usually offer in these cases: ## If you are doing manual testing you probably _want_ to automate things but you either don't have time to automate because you are too busy doing manual tests, or you might have people (not "resources" , "people"!) that do not have the automation skill yet: ### If you don't have time to automate [you are probably in Survival Mode. You are over committed](http://5whys.com/blog/definition-survival-mode.html). You will have to change your commitments and have a good hard talk with your peers and technical leadership about changing your commitments so you get more time to automate, otherwise you are in a vicious cycle that only leads to a worse place (the less time you have to automated the more you test manually, and so you get even less time to automate) ### If your people are missing the skills: [coach them and give them time to learn those new skills](http://5whys.com/blog/the-3-maturity-stages-of-a-software-team-and-how-scrum-fails.html) (or possibly hire an expert that teaches them) ## If you are using static environments \- [look into moving to Ephemeral environments](http://www.ibm.com/developerworks/library/a-devops3/): environments that are spun up and torn down automatically before and after a specific pipeline stage, such as testing. ## If you are configuring your static environments manually: Look into tools such as Chef Provisioner, Terraform or Puppet to automate the creation of those environments in a consistent way that is fully automated, and that also keeps configuration current. This also solves the issue of having no record of any configuration changes to environments, since all changes are in the form of changing a file in source control, and having the tool take care of the provisioning and configuration for you. Less human error, auditing, versioning: who wouldn't want that? It's the essence of infrastructure as code: treating our infrastructure the same way we treat our software: through code! This also gives you the ability to use tools to test your environments and configuration with tools such as Chef Compliance, InSpec and others. ## If you have organizational policies that prevent automated deployment: - Look into creating a **specialized, cross functional team** of people within the organization, that is tasked with creating a **new policy** that allows for some categories of application changes to be "pre-approved". - This will enables pipelines to deploy changes that fall into this category direction to production without going through a very complicated approval process that takes a long time. - For **other types of changes** automate as much as you can whatever compliance, security and approval checks (yes, even ServiceNow has APIs you can use, did you know?), and then put them into a special approval pipeline, that removes as much of the manual burden on the change committees as possible, so that the meetings they have become more frequent, and faster "we want to make this change, it has already passed compliance automated tests and security tests in a staging environment - are we good to go?" - For **changes that affect other systems** in the organization, you can look into creating a special trigger of those external pipelines that get deployed with your new system in parallel to a test environment, and then make tests are run on the dependent system to see if you broke it. Pipelines that trigger other pipelines can be an ultimate weapon to detect cross-company changes with many dependencies. I will expand on this in a later post. --- title: "DevOps Metric: Amount of Defects and Issues: definition." url: "https://pipelinedriven.org/article/devops-metric-amount-of-defects-and-issues-definition.md" canonical: "https://pipelinedriven.org/article/devops-metric-amount-of-defects-and-issues-definition" published: "2016-10-29T09:10:27.276000+00:00" updated: "2016-10-29T09:10:27.585000+00:00" tags: [] author: "Roy Osherove" description: "Important to : VP of QA, Operations Definition : How many bugs or tickets are found in production/ how many recurring bugs in production. As well as - how…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # DevOps Metric: Amount of Defects and Issues: definition. 29 October 2016 · 1 min read · Roy Osherove - **Important to**: VP of QA, Operations - **Definition**: How many bugs or tickets are found in production/ how many recurring bugs in production. As well as - how many deployment issues during release are encountered. - **How to measure**: tickets from customers, support records. For deployment issues: post release retrospectives. - **Expected Outcome**: Amount of deployment issues, defects and recurring defects should decrease as DevOps maturity grows. Many organizations make an almost-conscious choice: go fast, or build with high quality. Usually they lose on both ends. Measuring defects and deployment issues is a good measure for a DevOps maturity model. You should become faster while not compromising on quality, because quality gates are fully automated in the form of tests in multiple steps, instead of manual error prone steps. The same applies for error prone deployments: they almost stop happening, and when they do happen, the deployment script itself is fixed and can be tested before being used to deploy to production. Just like regular software development life cycles. The ROI is huge. --- title: "DevOps Metric: Frequency of Releases Definition" url: "https://pipelinedriven.org/article/devops-metric-frequency-of-releases-definition.md" canonical: "https://pipelinedriven.org/article/devops-metric-frequency-of-releases-definition" published: "2016-10-29T08:59:27.165000+00:00" updated: "2016-10-29T08:59:27.243000+00:00" tags: [] author: "Roy Osherove" description: "Important to : Release Manager, CIO Definition : How often an official release is deployed to production, in paying customer's hands How to measure : If no…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # DevOps Metric: Frequency of Releases Definition 29 October 2016 · 2 min read · Roy Osherove - **Important to**: Release Manager, CIO - **Definition**: How often an official release is deployed to production, in paying customer's hands - **How to measure**: If no pipeline: release schedule. Otherwise, history data for Pipeline Production deploy step. - **Expected outcome**: The time between releases should become shorter and shorter (in some cases an order of magnitude shorter) as DeOps maturity grows. It's one thing for the IT of an organization to create pipelines so fast that they can release at will. The other side of that coin is that of the business.  ``` If IT is no longer a bottleneck, it is the business' responsibility to decide *when* to release new versions. ``` In traditional organizations IT is usually too slow to supply and deploy the release as fast as the business requires, so the business is forced to operate at the speed of IT , which might mean 4 release a month, 4 releases a year or even less frequent.  In a DevOps culture, every code check in (A 'commit') is a potentially releasable product version, assuming it has passed the whole continuous delivery pipeline all the way up to the staging servers.  We can use the "Frequency of release metric to decide "how many releases do we want to do per year, month, week?" We can then measure our progress.  As IT becomes faster, we can cut down our scheduled release time. As the pipeline supports more and more automated policy and compliance checks as feedback cycles, more time is removed from manual processes and potential releases become easier to achieve. The question "_How often should we pull the trigger to deploy the latest potential release that has passed through the pipeline?_" shifts from becoming IT's problem to being owned solely by business stakeholders. As the schedules become more frequent, the competitiveness in the marketplace becomes better. In some organizations (Netflix, or Amazon are a good example), the decision on a release schedule is removed completely and only the pipeline's "flow" of releases that automatically get deployed determines the release frequency, which might happen as often as every minute or even less. --- title: "DevOps Metric: Mean time to recovery (MTTR) Definition and reasoning" url: "https://pipelinedriven.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning.md" canonical: "https://pipelinedriven.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning" published: "2016-10-29T08:51:46.104000+00:00" updated: "2016-10-29T08:51:46.194000+00:00" tags: [] author: "Roy Osherove" description: "Important to: VP Operations, CTO Definition :When a production failure occurs, how long does it take to recover from the issue? How to measure: Different…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # DevOps Metric: Mean time to recovery (MTTR) Definition and reasoning 29 October 2016 · 4 min read · Roy Osherove **Important to:** VP Operations, CTO **Definition**:When a production failure occurs, how long does it take to recover from the issue? How to measure: Different between systems. Common metric is production downtime avg over last ten downtimes   **Expected outcome:** MTTR should become lower and lower as DevOps maturity grows. **MTTR vs Mean time to Failure** To understand MTTR, we have to first understand its evil brother: **MTTF** or "Mean time to failure" which is used by many organizations' operational IT departments today. MTTF means "How much time passes **between** **failures** of my system in production?". **Why is MTTR more valueble?** There are two arguments in favor of putting MTTR as a higher priority than MTTF (we don't want to ignore that we fail often, but it's not as important as MMTR). **Perception**. If Amazon.com was down once every three years, but it took them a whole day to recover, consumers won't care that the issue has not happened for three years. All that will be talked about is the long recovery time. But if Amazon.com was down for 3 times a day for less than one second, it would barely be noticeable. **Wrong Incentives.** Let's consider the incentive MTTF creates on an operations department: The less often failure happens in the first place, the more stable your system is, the more bonus you get at the end of the year in your paycheck. > What's the best way to keep a system stable? Don't touch it! Avoid any changes to the system, release as little as possible, in a very controlled, waterfall-ish manner to make sure releasing is so painful that the person on the other end really really wants to release, and would go through all the trouble of doing so. Sounds familier? This "_Stability above all else_" behavior goes exactly _against_ the common theme of DevOps: release continuously and seize opportunities as soon as you can. The main culprit for breeding this anti-agile behavior is a systematic influence problem: What we measure influences people into doing the behavior that hurts the organization (You can read more about this in my book "[Elastic Leadership](http://elasticleadership.com/)" in the chapter about "Influence Forces" ([here is a blog post](http://5whys.com/blog/how-to-plan-and-influence-change-at-your-company.html) that talks about them in more details). Developers can't rest on their laurels and claim that operations are the only ones to blame for slowing down the continuous delivery train here. Developers have their own version of "Stable above all else" behavior which often can be seen in their reluctance to merge their changes into the main branch of source control (where the build pipelines get their main input from). Ask a developer if they'd like to code directly on the main branch and many in enterprise situations will tell you they'd be afraid of doing so in the fear of "breaking the build". Developers are trying to keep the main branch as "stable" as possible so that the version going off to release (which is in itself a long and arduous process as we saw before), has no reason to come back with quality issues. "Breaking the build" is the developer's version of "mean time to failure", and again, here the incentives from management are usually the culprit. If middle managers tell their developers it's wrong to break a build, then they are driving exactly the same fear as operations have. The realization that "builds are made to be broken" is a bit tough to swallow for developers who fear that they will hold up then entire pipeline and all other teams that depend on them. Again, the same thought here applies: Failure is going to happen, so focus on the recovery aspect: how long does it take to create a fix for something that stops the build? If you have automated unit tests, acceptance tests and environments, and you're doing test-driven development, fixing an issue that stops the pipeline can usually be a matter of minutes: code the fix, along with tests, see that you didn't break anything, and check it into the main branch. Both operations and development have the same fear: don't rock the boat. how does that fit in with seizing opportunities as quickly as possible? How does this support "mean time to change" to be as fast as possible? --- My answer is that measuring and rewarding MTTF above all else absolutely does not support a more agile organization. That fear of "breaking the build" and "keep the system stable" is one of the reasons many organizations fail at adopting agile processes. They try to force "release often" down the throat of an organization that is measured and rewarded when everything stays stable instead of being rewarded and measured on how often can you change and how fast can you recover if a failure occurs. --- MTTR is a very interesting case of Devops Culture taking something well established, and turing it on its head. If DevOps is about build, measure learn, and fast feedback cycles, then it should become an undeniable truth that "whatever can go wrong, will go wrong". if you embrace that mantra, you can start measuring "mean time to recovery" instead. The MTTR incentive can drive the following behaviors: - Build resilience into the operations system as well as the code. - Build faster feedback mechanisms into the pipeline so that a fix can go through the pipeline faster - Creating a system and code architecture that minimizes dependencies between teams, systems and products so failures do not propagate easily and deployment can be faster and partial - Add and start using better logging and monitoring systems - Create a pipeline to drivers a deployment of an application fix into product as quickly as possible. - Making it just as easy and fast to deploy a fix as it is to roll back a version With **mean time to recovery**, you're incentivizing people based on how much they contribute to the "concept to cash" value stream, or the "mean time to change" number. --- title: "DevOps Metric Definition: Mean time to change (MTTC) (vs change lead time)" url: "https://pipelinedriven.org/article/devops-metric-definition-mean-time-to-change-mttc-vs-change-lead-time.md" canonical: "https://pipelinedriven.org/article/devops-metric-definition-mean-time-to-change-mttc-vs-change-lead-time" published: "2016-10-29T08:34:47.556000+00:00" updated: "2016-10-29T08:36:52.386000+00:00" tags: [] author: "Roy Osherove" description: "Also called \"time from concept to cash\". Important to : Everyone, especially the CEO, CIO and CTO. Definition : How long does it take a new average feature,…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # DevOps Metric Definition: Mean time to change (MTTC) (vs change lead time) 29 October 2016 · 1 min read · Roy Osherove Also called "time from concept to cash". **Important to**: Everyone, especially the CEO, CIO and CTO. **Definition**:  How long does it take a new average feature, idea, fix or any other kind of change to get into a paying customer's hands, in production, from the moment of inception in someone's mind. MTTC is what it takes you from the moment you see an opportunity until you can actually utilize it. The faster MTTC is, the faster you can react to market changes. **How to measure:**  We start counting from moment of the change's inception in someone's head (Imagine a marketing person coming up with a competing idea to that of a competitor's product, or a bug being reported by a customer) One way to capture and measure mean time to change is by doing a value streaming exercise, as we will touch on in a later chapter in this book. **Expected Outcome:**  MTTC should become shorter and shorter as DevOps maturity grows.  **Common Misunderstandings:**  MTTC is not the same as the often cited "**Change lead time**" as proposed in multiple online publications. Change Lead time (at least as far as I could see) only counts the time from the start of development of a feature, when real coding begins. MTTC will measure everything that leads up to the coding as well, which might include design reviews, change committees, budgeting meetings, resource scheduling and everything that stands in the way of an idea as it makes its way into the development team, all the way through to production and the customer. From a CEO, CIO and CTO view , MTTC is one of the most important key metrics to capture. Unfortunately, many organizations today do not measure this. In many companies I've worked with, MTTC was anywhere from one month to twelve months. --- title: "What is Enterprise DevOps?" url: "https://pipelinedriven.org/article/what-is-enterprise-devops.md" canonical: "https://pipelinedriven.org/article/what-is-enterprise-devops" published: "2016-10-29T00:51:34.028000+00:00" updated: "2016-10-29T09:53:23.847000+00:00" tags: [] author: "Roy Osherove" description: "To understand Enterprise DevOps, we have to first define what \"regular\" DevOps is. Here is my current definition of DevOps. Let's call it \"DevOps 0.1\" What…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # What is Enterprise DevOps? 29 October 2016 · 1 min read · Roy Osherove To understand Enterprise DevOps, we have to first define what "regular" DevOps is. Here is my current definition of DevOps. Let's call it "DevOps 0.1" ### What is DevOps? (Version 0.1) DevOps is a culture that drives and enables the following behaviors: - Treating Infrastructure and code as the same thing: Software - That Software is built and deployed to achieve fast feedback cycles - Those feedback cycles are achieved through continuous delivery pipelines - Those pipelines contain the automated coded policy of the organization  - Those pipelines are as human free as possible - Humans share knowledge and work creatively with work centered around pipelines  - Knowledge silos are broken in favor of supporting a continuous pipeline flow DevOps is the Agile operating model of modern IT, and is centered around the idea of a continuous delivery pipeline as the main building block of an Agile organization. ## DevOps Metrics DevOps is based on lean development concepts, as well as the lean startup movement's ideas. As such, the "**build, measure, learn**", the scientific method mantra, is very relevant to DevOps. That's why metrics play a very important role in a DevOps culture. They help determine gradual success over time in DevOps transformations. Here are the main metrics being used in DevOps based organizations today: 1. [Mean time to change](https://pipelinedriven.org/article/devops-metric-definition-mean-time-to-change-mttc-vs-change-lead-time) 2. [Mean time to recovery](https://pipelinedriven.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning) 3. [Frequency of Release](https://pipelinedriven.org/article/devops-metric-frequency-of-releases-definition) 4. [Defect Rates](https://pipelinedriven.org/article/devops-metric-amount-of-defects-and-issues-definition) --- ## What is Enterprise DevOps? Enterprise DevOps is the application of DevOps values in an environment that contains any of the following: - Many inter-dependent and related systems and sub systems, software and teams that rely on each other - Monolithic systems and/or static software/hardware environments - Lengthy Approval Gates or change control processes - Security, financial medical or other compliance requirements - Lengthy waterfall based processes among multiple teams and stake holders - Workflows that are mostly manual and error prone across multiple teams or stakeholders In order to create a pipeline-centered organization that is able to seize opportunities in the marketplace quickly. --- --- title: "Interview on Devops and Cloud Infrastructure" url: "https://pipelinedriven.org/article/2016/1/7/interview-on-devops-and-cloud-infrastructure.md" canonical: "https://pipelinedriven.org/article/2016/1/7/interview-on-devops-and-cloud-infrastructure" published: "2016-01-07T15:26:46.385000+00:00" updated: "2016-01-08T09:27:08.652000+00:00" tags: [] author: "Roy Osherove" description: "I've had the pleasure of being part of a panel on Devops and various patterns and anti patterns relating to cloud operations and continuous delivery. You…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Interview on Devops and Cloud Infrastructure 7 January 2016 · 1 min read · Roy Osherove I've had the pleasure of being part of a panel on Devops and various patterns and anti patterns relating to cloud operations and continuous delivery. You can listen to the podcast (called "Continuous Discussions" hosted by [Electric Cloud](http://electric-cloud.com/powering-continuous-delivery) ) on a special link [over on this page.](http://electric-cloud.com/blog/2015/12/continuous-discussions-c9d9-episode-31-recap-cloud-resources-and-your-cd-pipeline/) Enjoy! --- title: "Devops is..." url: "https://pipelinedriven.org/article/2015/10/29/devops-is.md" canonical: "https://pipelinedriven.org/article/2015/10/29/devops-is" published: "2015-10-29T08:31:21.990000+00:00" updated: "2015-10-29T08:31:22.627000+00:00" tags: [] author: "Roy Osherove" description: "Infrastructure & Code Treated as software, where Software Is created with fast feedback cycles, where Cycles Do not require monkey work by people, where…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Devops is... 29 October 2015 · 1 min read · Roy Osherove **Infrastructure & Code** Treated as software, where **Software** Is created with fast feedback cycles, where **Cycles** Do not require monkey work by people, where **People** Share knowledge, and work creatively, with repetitive work automated --- title: "BBBCTC - Branching, Binaries, Builds, Coding and Testing Culture" url: "https://pipelinedriven.org/article/2015/5/30/bbbctc-branching-binaries-builds-coding-and-testing-culture.md" canonical: "https://pipelinedriven.org/article/2015/5/30/bbbctc-branching-binaries-builds-coding-and-testing-culture" published: "2015-05-30T20:31:49.899000+00:00" updated: "2015-05-30T20:43:12.325000+00:00" tags: [] author: "Roy Osherove" description: "As part of a major drive to push for faster \"time to main\" you might need to change things in the following five areas (that end up affecting how long…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # BBBCTC - Branching, Binaries, Builds, Coding and Testing Culture 30 May 2015 · 3 min read · Roy Osherove As part of a major drive to push for [faster "time to main"](http://www.beautifulbuilds.com/blog/2015/5/15/time-to-main-measuring-continuous-integration) you might need to change things in the following five areas (that end up affecting how long people take to get their code checked into "trunk": ## Branching Culture Branches means merges, and merges take time. Branching culture in many places also mean "only merge when you feel you won't break things for anyone else" which effectively means many places use branches to "promote" code to the next stage of stability. This in turn means people might spend a lot of time "stabilizing for others" code before moving it onto the next branch (system testing the code locally, really). Less branches - less time to main! ## Binaries Sharing Culture When working with multiple teams that depend on each other's outputs, moving binaries between the teams becomes a time consuming challenge. I've seen teams use shared folders, emails and even Sharepoint pages to share their binaries across development groups. Find a simple way for teams to share and consume binaries easily with each other, and you've saved a world of hurt and time for everyone involved (for example: [dependency stash](https://leanpub.com/build)). ## Builds If you're going to reduce the amount of branches, you probably will need gated builds to make sure people don't check in code that breaks compilation on the main branch (trunk). This would be the first level of a build (gated build per module). the second level would possibly an hourly (or faster) integration build that also runs full systematic smoke tests and other type Also, if an integrated build does break, you will need to put fixing a red build as high priority for those who broke it, or those who have to adapt, before moving on to new features. One more: you will probably need a  cross-group build triage team that notices red builds as they  occur and has the ability to find out what is causing a red build, and where to redirect the request-to-fix toward. ## Testing Culture You will need to automate a bunch of the manual testing many groups so they can run it in a gated build, and for the smoke and system tests at the integrated build level. ## Coding culture Developers will need to learn and apply [feature toggles (code switches)](http://www.beautifulbuilds.com/feature-toggle-frameworks-list/) or learn how to truly work incrementally to support new features on the main trunk, that take longer than a day or so to build, without breaking everyone's code. There are two types of breakages that can occur on the main trunk (assuming gated builds took care of compile and module level unit and smoke tests): -  **Unintended breakages** : can usually be fixed easily directly on the trunk by the group that broke them, or by another group that needs to adapt to the new code. -  **Intended breakages.**   These can be broken up into: - **short-lived** intended breakages (such as group x has a new API change, they check it into main and expect group B to get it from main, adapt to it with a simple code change and check their stuff back into Main . this should usually be within an hour's work). - **Long lived** intended breakages: you're working on a feature for the next month or so. In that case you either work fully incrementally on that new feature, or you enable [feature toggles](http://www.beautifulbuilds.com/feature-toggle-frameworks-list/) (a.k.a 'branch by abstraction'). Either way we are talking about teaching developers a new culture of coding. Without this culture coding change , developers will always be afraid of working directly on main and you'll be back to private branches with everything they entail (a.k.a - 'long time to main'). --- title: "Time to Main - Measuring Continuous Integration" url: "https://pipelinedriven.org/article/2015/5/15/time-to-main-measuring-continuous-integration.md" canonical: "https://pipelinedriven.org/article/2015/5/15/time-to-main-measuring-continuous-integration" published: "2015-05-15T03:41:19.987000+00:00" updated: "2015-05-15T05:20:54.405000+00:00" tags: [] author: "Roy Osherove" description: "A.k.a \"Time to Trunk\". How do you measure your progress on the way to continuous integration(we're not even discussing delivery yet)? How do you know you…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Time to Main - Measuring Continuous Integration 15 May 2015 · 2 min read · Roy Osherove A.k.a "Time to Trunk". How do you measure your progress on the way to continuous integration(we're not even discussing delivery yet)? How do you know you are progressing towards the goal incrementally? Assuming you're in an enterprise situation, showing a KPI over time can help drive a goal forward. One of the KPIs I've used on a project was "Time to Main".  Let's start with the basic situation we'd like to fix, assuming multiple teams with bad architecture that causes many dependencies between the teams. See below chart: [![](https://pipelinedriven.org/img/c/image-asset-76c7de.png)](https://pipelinedriven.org/img/c/image-asset-76c7de.png "View full size") In the chart above we have four teams and one main branch. Each team also has it's own private branch(es) structure. A usual cycle of crating an application version is:  - new NEO binaries distributed to the other teams. HAMMER is adapted and then send to the ORCA team. ORCA team produces binaries to the DAGON team which then produce ORCA+HAMMER+DRAGON components to the ADAM team. Finally, they all merge to the main branch, in specific order. Obviously, not the best situation in the world. There are many things we can do to make things better, but first, we need a way to measure that we ARE making things better.  One measure I like to use is "Time to Main" . How long does  it take for a FULL version of all components that work with each other to get from developer's hands to MAIN? In our case the answer starts with: > 1 Day Hammer work + 3 days ORCA work + 3 days DRAGON work +  5 days ADAM work = 12 Days to Main. But wait , there's more. Many of the back and forth lines are actually branch merging actions. For large code bases, add a few hours per merge, which can add up to 2-3 days if not more. Total could be 14-16 days to Main, starting with team NEO publishing a new feature( in this case they are not in the same source control - as they are in a different company). > Note: > We are assuming that the timings mentioned here are "friction time" only. Time that is not used for code development, but the absolute necessary time developers need to feel confident to merge to MAIN . This might mean they run a bunch of manual or automated tests, debugging, installs. anything. It is time that will be spent whether there is or isn't major functionality being introduced, because the integration with the other components has a cost. There are many things we can choose to do here: - Ask all teams to work directly on main - Create a "Dependency cache" from the continuous build on "Main" so everyone can access the binaries. - Reduce manual testing with automation - Etc.. No matter what we do, we can always ask ourselves "How does this affect time to main?" We can always measure that tie, and see if we are actually helping the system, or just implementing local efficiencies while Time to Main remains the same, or even increases. From the [Theory of Constraints,](http://en.wikipedia.org/wiki/Theory_of_constraints) we can borrow the "Constraint" term. _Time to main_ is our _Constraint_ of actually having a shippable product increment we can demo, install or test at the system integration level. The more teams choose to use internal branches, the more time code has to wait until it sees "Main". Now we have a number that tells us if we are getting better or worse in our quest for continuous integration. --- title: "Pipeline Disintegration (Post Build Pipeline)" url: "https://pipelinedriven.org/article/2015/3/29/pipeline-disintegration-post-build-pipeline.md" canonical: "https://pipelinedriven.org/article/2015/3/29/pipeline-disintegration-post-build-pipeline" published: "2015-03-29T05:18:33.939000+00:00" updated: "2015-03-29T05:18:32.107000+00:00" tags: [] author: "Roy Osherove" description: "Other Names: Post Build Hand off, Time Capsule, Strangler Application Symptoms: It is very complicated to add features to the current build process, because…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Pipeline Disintegration (Post Build Pipeline) 29 March 2015 · 2 min read · Roy Osherove [![](https://pipelinedriven.org/img/c/image-asset-861558.png)](https://pipelinedriven.org/img/c/image-asset-861558.png "View full size") ### Other Names: Post Build Hand off, Time Capsule, Strangler Application ## Symptoms: It is very complicated to add features to the current build process, because the current tooling deficiencies, process bureaucracy or the responsible team is reluctant to add those features for their own reasons. ## Problem: The current build process is blocking from adding features required for the business to succeed, such as increasing feedback, adding more information, or adding more actions to ease manual work. Continuing to bang on that specific door might be a Sisyphean task, and spending too much time on it would be very unproductive and ineffective. ## Forces: You'd like to get more feedback sources to the build, but that might interfere with the team's current processes, area of responsibility, or, in regulated environments, even traceability, documentation and workload. You and other stakeholders believe that adding those features will increase feedback, quality or other benefits, but tacking new features on the current build seems to be stepping on everyones toes. ## Solution: Create a new pipeline that receives a _Time Capsule_ from the current build process. This _Time Capsule_ will contain the end artifacts from the build process (usually the "release" binaries and other support files, and possibly all the source code). Now that the time capsule is in your possession, you can put it through a new, separate pipeline that adds the steps that you might care about. Examples might include: - Static code analysis - Running unit tests (if those couldn't be run in the original build) - Deployments ### A more explicit example: 1. Make sure the "old" build copies or "drops" all related artifacts into a shared folder that will be accessible by the new pipeline. This should happen automatically every time the old build runs, which hopefully will be once or more a day. 2. Install an instance of a CI server of some sort (let's say it's TeamCity for the sake of argument), and create a new project for the new pipeline. This will hopefully be on a seperate server so as not to interfere with the old build's require hardware and software resources. 3. The first action as part of the new pipeline will be, instead of using source control, copying the time capsule binaries into the local build agent that will be invoked by the new CI tool during the run. 4. Next steps might then include running unit tests, static code analysis or anything else your heart desires, without interfering with the old build structure. 5. Notify stakeholders if the new pipeline fails. --- title: "Split and Parallelize" url: "https://pipelinedriven.org/article/2014/10/3/q5b4rm6abtjxeleaj0xfapo1n3khnl.md" canonical: "https://pipelinedriven.org/article/2014/10/3/q5b4rm6abtjxeleaj0xfapo1n3khnl" published: "2014-10-03T19:52:57.611000+00:00" updated: "2014-10-03T19:53:59.337000+00:00" tags: [] author: "Roy Osherove" description: "Other Names: Split Work Symptoms: The build is taking too long Problem: It could be that one or more tasks or build steps that are comprised of many actions…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Split and Parallelize 3 October 2014 · 3 min read · Roy Osherove ### Other Names: - Split Work ## Symptoms: - The build is taking too long ## Problem: - It could be that one or more tasks or build steps that are comprised of many actions are taking too much time. For example: running 1000 regression tests (each taking 1-4 minutes) is taking 24 hours, and this is slowing down the feedback rate from the build. ## Solution: Split the step into several parallel running steps, each running a part of the task. The build step will then theoretically be able to be no longer than the longest time it takes to run just one of the split parts in parallel. The solution requires: - having multiple build agents, or workers, to be able to run things in Parallel - The split parts need to be somewhat similar in size to gain speed improvement - the amount of split parts should be at least as the amount of how many agents will be able to run each split part. ## Example Say we have 1000 regression tests and one build step called "Run Regression Tests". ## Step 1: Split the tests to runnable parts We split the regression tests into separate runnable "ranges" or categories that can be executed separately from the command line. There are many ways to do this: Split the tests to multiple assemblies with names ending with running numbers, put separate "categories" on different tests, so you can tell the test runner to run only a specific category, and more. It is important that the size of the "chunks" you split into will be somewhat similar, or the build speed gains will be less optimal. In our case, we have 1000 tests, and 10 agents the test chunks can run on. So we create 10 test categories named "chunk1", "chunk2" etc.. An even better scheme would have been if our tests could be split across 10 different logical ideas. Say we are testing our product with 10 different languages, approx. 100 tests per language. It would have been perfect to split them based on language and name each chunk based on that language. Later on, this also makes things more readable at the build server level. If you can only find 2 or 3 logical "categories" for the tests to split based on, (say "db tests", "ui tests" and "perf tests") you might want to go ahead and split each category into chunks to make things faster if you have more than 3 agents. For example "ui-chunk1", "ui-chunk2" until you reach at least the number of agents you have. ## Step 2: Create a parallel test run hierarchy in the CI server If you're using TeamCity, you would now create the following: - new sub project called "run regression tests" - 10 steps (assuming you have 10 build agents that can run them) , one for each test chunk: - Run UI Tests Chunk 1 (A command line step calling the test runner command with the name of the category to be run) - Run UI Tests Chunk 2 (same command line, different category) - Run UI Tests Chunk 2 - Run DB Tests Chunk 1 - ... - Trigger all Regression Tests Note the last build step. This build step is merely an empty build step that has a snapshot dependency to all the listed "chunk" steps. when triggered, it should (assuming you have multiple agents enabled and ready to run these) run all the chunks in parallel, which will finish as quickly as it takes the longest running chunk to finish. --- title: "Irrelevant Build" url: "https://pipelinedriven.org/article/2014/10/3/-irrelevant-build.md" canonical: "https://pipelinedriven.org/article/2014/10/3/-irrelevant-build" published: "2014-10-03T19:49:49.642000+00:00" updated: "2014-10-03T19:54:16.601000+00:00" tags: [] author: "Roy Osherove" description: "Symptoms: The team doesn't seem to care that one of the builds is failing A build is red, but might not be failing. Only one person from a different team or…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Irrelevant Build 3 October 2014 · 2 min read · Roy Osherove ## Symptoms: - The team doesn't seem to care that one of the builds is failing - A build is red, but might not be failing. Only one person from a different team or role can tell whether it actually failed or not. ## Problem: One possible cause of is might be that the failing build is not essentially relevant to the current team, or the build result is not understandable (see "binary result" in that case). ## Solution: Remove the failing build from the visualization screen visible to all the team. By keeping the build there the team will eventually stop caring about "seeing red" on the screen. The red color has to mean something. ## Example: In one of the projects I was involved with, there was a big screen up on the wall that showed the status of all the builds. Some of the builds were green, but there were a couple of the builds that related to QA regressions and UI testing that were practically always red. Because there was a lack of good communication between the QA and dev teams, none of the developers really understood what is wrong with that build, or even cared. What's more, the QA lead would later tell me that a red build in QA does not necessarily mean a failed build. Since UI tests were very fragile, a statistical analysis of sorts was used, so that if X builds of a certain number passed, it was considered a success. The problem was that the developers were too used to seeing red, and when they saw red they were very used to thinking "not my problem". The first thing I did was remove the always failing builds from the dev wall, so that all the visualized builds would be relevant to the people watching them. Suddenly there was a green hue in the room as all the builds were passing. Developers would go by and ask me in amusement, "What is that weird color?" and "what's wrong with the build? it's green!" Then, when the builds failed due to dev issues, developers had more reasons to care. I would only show the QA builds again if they were more stable or if there was better cross team communication so that devs could help QA fix the builds. As long as there was nothing they could do about it, they should only see red when it pertains to work they can control. --- title: "Parallel Firehose" url: "https://pipelinedriven.org/article/2014/9/22/parallel-firehose.md" canonical: "https://pipelinedriven.org/article/2014/9/22/parallel-firehose" published: "2014-09-22T05:02:21.704000+00:00" updated: "2014-09-22T05:02:22.280000+00:00" tags: [] author: "Roy Osherove" description: "Pattern: Parallel Firehose Other Names: Build Environment Per Stakeholder Symptoms: You can't deploy a new version of the application to your test or…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Parallel Firehose 22 September 2014 · 2 min read · Roy Osherove ## Pattern: Parallel Firehose ### Other Names: - Build Environment Per Stakeholder ## Symptoms: You can't deploy a new version of the application to your test or staging environment because it's in use by other people in the organization such as the QA team, Product owners, or customers. So you can only deploy during the night time, or some other constraint. ## Problem: Developers are sharing the same environment (machines, physical or virtual) with other stakeholders. ## Forces: - You want to do continuous delivery, without needing to wait for others. - Other stakeholders want to use or test the application without needing to stop, or be annoyed by your deployments making the application slower or crashing. ## Solution: Create a separate environment for your other stakeholders, that they can use without you interfering with it, and they can pull a new version to at will. ## Example: Assuming you have the following environment: - A Build Machine that runs TeamCity or Jenkins - A Build Agent Machine (Build1) that runs a TeamCity agent, or a Jenkins Worker - A Test Machine (Test1) where BUILD1 deploys your application to for demo purposes and for use by QA - A developer machine (DEV1) that keeps checking in code to the main branch You want to do continuous testing by running unit and integration tests on check in, as well as smoke tests and acceptance tests by deploying the application automatically on every check in. You can't because QA are using the machine during work hours. You can only deploy and run those tests at night, and they might actually break whatever nightly tests the QA folks are running. Here is one way to solve this: Setup the environment with new machines so you end up with an environment like this: - A Build Machine (BUILDMASTER) that runs TeamCity or Jenkins - A Build Agent Machine (Build1) that runs a TeamCity agent, or a Jenkins Worker - _A Build Agent Machine (Build2) that runs a TeamCity agent, or a Jenkins Worker_ - A Test Machine (Test1) that an agent deploys your application to for dev use - _A Test Machine (Test2) that an agent deploys your application to for QA use_ - A developer machine (DEV1) that keeps checking in code to the main branch Then, in your CI server setup the following two workflows: ### workflow 1: Fully automated Pipeline - Triggered on Check in to main branch - compile on a free agent - Run unit tests on a free agent - Run Integration tests on a free agent - Deploy to TEST1 on a free agent - Run acceptance tests only on TEST1 ### workflow 2: Pull Based QA Pipeline - Triggered by QA clicking on a button, or on a set schedule, or both - compile using a free agent - Run unit tests using a free agent - Run Integration tests using a free agent - Deploy to TEST2 using a free agent - run QA tests on TEST2 ## Summary With this approach nobody is stepping on anyone's toes: - Developers can do continuous tests on TEST1 dozens of times a day (yes, some teams actually achieve this). A.k.a it's so fast it's like "drinking from a fire hose" (hence the pattern name) - QA get a nice stable environment they can control and pull versions into. You can repeat this process for new stakeholders. If you're using cloud services, it's even easier to setup such new environments. --- title: "Build Pattern: Deploy by Proxy - Added to the book" url: "https://pipelinedriven.org/article/2014/7/26/build-pattern-deploy-by-proxy-added-to-the-book.md" canonical: "https://pipelinedriven.org/article/2014/7/26/build-pattern-deploy-by-proxy-added-to-the-book" published: "2014-07-26T21:42:31.265000+00:00" updated: "2014-07-26T22:23:33.487000+00:00" tags: [] author: "Roy Osherove" description: "It's been a couple months of other work. but there's finally new material in the book and more coming soon. First: I've now grouped the chapters under…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Build Pattern: Deploy by Proxy - Added to the book 26 July 2014 · 1 min read · Roy Osherove It's been a couple months of other work. but there's [finally new material in the book and more coming soon.](http://leanpub.com/build) - First: I've now grouped the chapters under "parts", that start telling a story, grouping the patterns under the following parts: Introduction,Separation of Concerns, Productivity, Maintainability, Team Collaboration, Stakeholders, and Branching. - I've also added a new pattern under stakeholders called "Deploy by Proxy" about dealing with Security Folks who won't let you deploy to production. - Also made slight adjustments to "Fill in the blanks" to make it clearer relating to separation of concerns topic. - On a related note, I've published a video course on beautiful builds which contains this information and more that will be added to the book plus demos with teamcity.  You can grab it at http://courses.osherove.com . it costs $25 until the end of July. --- title: "Use Version Aware Build Scripts" url: "https://pipelinedriven.org/article/2014/3/2/use-version-aware-build-scripts.md" canonical: "https://pipelinedriven.org/article/2014/3/2/use-version-aware-build-scripts" published: "2014-03-02T21:26:30.083000+00:00" updated: "2014-03-02T21:29:49.342000+00:00" tags: [] author: "Roy Osherove" description: "I am thinking of writing a small booklet about better builds . this blog post might end up as a part of it . Let me know if you’re interested by taking a…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Use Version Aware Build Scripts 2 March 2014 · 3 min read · Roy Osherove > I am thinking of writing a small [booklet about better builds](https://leanpub.com/build). this blog post might end up as a part of it[. Let me know if you’re interested by taking a look here.](https://leanpub.com/build) While I do love to use Continuous Integration Servers such as teamcity, Jenkins and more, I try to use them as much as possible as “dumb triggers” to my build actions. They do one thing and one thing only, and that is to trigger the correct build script with the correct resources (source version, artifacts form other builds, etc), and nothing more.  I don’t add any custom build actions such as compilation, source checking etc, because I believe that the build actions themselves have a very strong connection with the correct version of the source. For example, at an earlier version of the source you might need to compile a specific set of projects, and in future source versions, maybe new projects were added, some were removed, or renamed etc.  If you have your build scripts as part of your source control, and versioned along with the source, they will always match the current source version. So what do those processes trigger? A script that is inside source control, tied directly to the current project\`s source committed version. It might be an xml file (ugh), or a rake file, or a finalbuilder script file (my favorite). Then it becomes easier to revert to an older build version to recreate it, or to test differences between versions, or to build multiple different branches of the same applications in parallel for different clients. See if you can take your current CI build, and have it run on a version of the source from 6 months ago. How many changes will that require of you to make it work? If your build is tied to source version, things should feel much simpler. ### Single Responsibility To me it is also about the single responsibility principle.  A Continuous Integration server should do one thing well – manage builds, their resources, and their scheduling.  It should not also need to worry about what is inside each build. In uncle bob’s [newspaper metaphor](http://www.markhneedham.com/blog/2008/09/15/clean-code-book-review/), the builds would be like different sub headings. Each does one thing, and the caller of those functions would only call them, without caring what they do. Thinking about it the other way around, if you have your whole build script running inside one large set of teamcity custom actions, all managed in the browser, and then you refactored that script to separate script files that are part of the source code, you’d be closer to what I am trying to do. So when I see companies boasting about how many custom tools their CI server can do, and how you can manage it all in the browser easily, I remind myself that the more I use those custom build features without the context of a current source version they are attached to, the more work I will have to do to change them every time I need to go back and forth between versions. ###  Version Aware Artifact Paths Following that line of thinking, I might argue that version aware build script “cubes” in source control should also be internally-deciding on what artifacts the encompassing CI process that is running them will be exposed to, and allowed to share with other builds. Today such things are managed solely at the CI tool level, but it would be good if the tools enabled artifact paths to be tied to the current version of the source. --- title: "Avoid XML Facing Build Tools" url: "https://pipelinedriven.org/article/2014/3/2/avoid-xml-facing-build-tools.md" canonical: "https://pipelinedriven.org/article/2014/3/2/avoid-xml-facing-build-tools" published: "2014-03-02T21:25:55.049000+00:00" updated: "2014-03-02T21:26:07.762000+00:00" tags: [] author: "Roy Osherove" description: "Having your build scripts in XML is one of the worst things you can do for maintainability and readability. Treat your build scripts like you treat your…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Avoid XML Facing Build Tools 2 March 2014 · 1 min read · Roy Osherove Having your build scripts in XML is one of the worst things you can do for maintainability and readability. Treat your build scripts like you treat your source code (or, how developers **should** treat their source code) – with respect for readability and maintainability, not just for performance. To that end, XML is a bad candidate to have. If you have any more than 15 or 20 lines of xml in a build document: - It is hard to tell where things start, and what they depend on - It is hard to debug the build - It is hard to add or change things in the build logic - XML case sensitivity sucks - Creating, Adding and using  custom actions is a chore To avoid XML, you can start using some of the many build tools out there that are either visual, or support a domain specific language for builds that is more readable, maintainable or debuggable than XML: - [Rake](http://rake.rubyforge.org/) – Ruby Based DSL (Domain Specific Language) for builds, that is very robust and quite readable - [FinalBuilder](http://www.finalbuilder.com/) and [VisualBuild](http://www.kinook.com/VisBuildPro/) are two visual build scripting tools that give great readability into your build script --- title: "Start Products with an Empty Build and Deploy Pipeline" url: "https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline.md" canonical: "https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline" published: "2014-03-02T21:25:14.758000+00:00" updated: "2014-03-02T21:25:33.094000+00:00" tags: [] author: "Roy Osherove" description: "Builds can be nasty beasts when It’s finally time to organize and do them. There are so many things you did not think about, that it can take days or weeks…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Start Products with an Empty Build and Deploy Pipeline 2 March 2014 · 2 min read · Roy Osherove Builds can be nasty beasts when It’s finally time to organize and do them. There are so many things you did not think about, that it can take days or weeks to come up with a build for an existing system. To avoid the nastiness, start with an empty build, on the first day of your project. Then grow it gradually. Before you start your first feature on a product that you know needs to exist for a long time, start with creating the pipeline for deploying it. - Create an empty source project. Something that is a stand-in for the real project you are about to develop. If it is a web application, it could be a single page with “hello world” on it. This project will be what you throw through your empty pipeline to test it. - Create a simple CI build script [that will live with the code](https://pipelinedriven.org/article/2014/3/2/use-version-aware-build-scripts). For now it might only compile your empty project. Make sure it is relative to the code location so you can run it anywhere. - Create a simple “Deploy to production” build script that lives with your code. For now this will only copy your project or put some file somewhere on a production server. - Oh, you should have a production server to deploy to, so you can deploy your product. that is step #1  of the first iteration! - Create a new project in your CI system (I use teamcity, but go ahead and use Jenkins or anything else you desire. ) In that project create a CI build configuration that triggers the CI build script, and a Deploy to Prod Configuration that triggers the Deployment build script you wrote. - Connect the CI server to your source code repository, and make it trigger the CI on commit. - Make the CI artifacts available to the Deployment configuration. - Make the Deployment Configuration trigger automatically on a successful CI build. - Run the whole thing and see a “hello world” in production **Now you are ready to nit pick:** - You might want to create a “deploy to test” configuration and script that triggers instead of production. - You might want to do the same with a **staging** environment ## Now you have an empty build and deploy pipeline. You can start writing real code, and as real code gets created, you only need to start doing small modifications to your build scripts , or CI process variables to make things work nice and smoothly. Now the build grows and flourishes alongside the source code, instead of as an afterthought. and it is remarkably easier to handle because the changes are much smaller. --- title: "Rolling Builds and the Plane of Confidence" url: "https://pipelinedriven.org/article/2014/3/2/rolling-builds-and-the-plane-of-confidence.md" canonical: "https://pipelinedriven.org/article/2014/3/2/rolling-builds-and-the-plane-of-confidence" published: "2014-03-02T21:24:24.850000+00:00" updated: "2014-03-02T21:24:49.162000+00:00" tags: [] author: "Roy Osherove" description: "When I check in code I have been working on, I feel: 1. I want to wait for the build result 2. I don’t want to waste time waiting for the build results,…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Rolling Builds and the Plane of Confidence 2 March 2014 · 2 min read · Roy Osherove When I check in code I have been working on, I feel: 1.    I want to wait for the build result 2.    I don’t want to waste time waiting for the build results, doing nothing 3.    But I fear doing something in the code until I’m sure I didn’t break anything Supposedly, this is where a CI build configuration comes into place. CI builds are supposed to be as fast as possible so we can get feedback about what we checked in as quickly as possible and get back to work. But the tradeoff is that CI builds also do fewer things so that they can become faster, Thus leaving you with a sense of some risk, even if the build passed. This is why I like to have “**Rolling Builds**”. I like to have builds trigger each other in the Continuous Integration Server: •    A Check-in triggers the CI build •    A successful CI build triggers a nightly build. •    A successful nightly build triggers a deploy to test build. I think of the builds now as single waves crashing on my shoreline. Each build is a slightly bigger wave. Each wave crashing on the shore brings with it a layer of confidence in the code. And because they happen serially, I can choose to just relax and watch the waves of increasing confidence crash on the shore, or I can choose to continue coding right after the first wave of confidence. As I program, the next waves of build results hit the shoreline, and my notification tray tells me what that wave brought with it. Another “green” result wave tells me to go on about my business. A “red” wave tells me to stop and see what happened. But I always wait for at least the first wave to come ashore and tell me what’s going on. I need that little piece of information that tell me “seems legit so far” so I can feel 50@5 good about going back to coding. If my changes were big, I might wait until the next wave to see what to do. So my confidence after check in is not a black or white result. It is a continuous plane of increasing confidence on the code that I am writing, that is topped off when the code is deployed to production. > **Note**: This text is part of a [“Beautiful Builds” Booklet](http://beautifulbuilds.com/) I am working on. --- title: "How to make your builds FAST using DRY and SRP Principles" url: "https://pipelinedriven.org/article/2014/3/2/how-to-make-your-builds-fast-using-dry-and-srp-principles.md" canonical: "https://pipelinedriven.org/article/2014/3/2/how-to-make-your-builds-fast-using-dry-and-srp-principles" published: "2014-03-02T21:23:20.818000+00:00" updated: "2014-03-02T21:24:04.796000+00:00" tags: [] author: "Roy Osherove" description: "(this will become part of my upcoming booklet on beautiful builds ) One of the things that kept frustrating me when I was working on various types of build…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # How to make your builds FAST using DRY and SRP Principles 2 March 2014 · 3 min read · Roy Osherove > _(this will become part of_ [_my upcoming booklet on beautiful builds_](http://beautifulbuilds.com/)_)_ One of the things that kept frustrating me when I was working on various types of build configurations in our CI server, was that each build in a rolling wave would take longer than the one before it. The **CI build** was the fastest, the **nightly build** was slower, because it did all the work of CI (compile, run tests) plus all the other work a nightly needed (run slow tests, create installer etc..). The **Deployment to test** build would maybe just do the work of the CI and then deploy, or, in other projects, would do the work of nightly, then deploy, because wouldn’t you want to deploy something only after it has been tested? And lastly, **production deploy builds** took the longest, because they had to be “just perfect”. so they ran everything before doing anything. That sucked, because at the time, I did not discover one of the most important features that my current CI tool had : the ability to share **artifact**s between builds. Artifacts are the output of the build actions. these could be binary compiled files, or they could be configuration files, or maybe just a log file, or just the source files that were used in the build , from source control. ## **Realization** Once I realized that build configurations can “publish” artifacts when they finish successfully, and that other build configurations can then “use: those artifacts in their own work, things started falling into place. I no longer needed to re-do all the things in all the builds. Instead, I can use the [DRY](http://en.wikipedia.org/wiki/Don't_repeat_yourself) (Don’t repeat yourself) principle in my build scripts [(remember that build scripts are kept in source control, and are simply executed by a CI build configuration that provides them with context, such as environment parameters, or the artifacts from a previous build).](https://pipelinedriven.org/article/2014/3/2/use-version-aware-build-scripts) Instead, I can make each [rolling wave build](https://pipelinedriven.org/article/2014/3/2/rolling-builds-and-the-plane-of-confidence), only do one small thing [(single responsibility principle](http://en.wikipedia.org/wiki/Single_responsibility_principle)), that gets added on top of the artifacts shared by the build before it. ## **For example:** - The CI Build Only gets the latest version from source control, compiles in debug mode, and runs the fastest tests. Then it **published the source and compiled binaries for other builds to use later on**. Takes the same amount of time as the previously mentioned CI build. - The Nightly build gets the artifacts from the latest successful CI Build, and only: compiles in RELEASE mode, runs the slow tests, creates installers, and **publishes the installer, the source, and the binaries for later builds**. Notice how it does not even need to get anything from source control, since those files are already part of the artifacts (depending on the amount of source this might not be a good idea to publish source artifacts due to slowness, but that depends on the size of the project). takes half the time of the previously mentioned nightly build. - The Deploy (to test) build, gets the installer from the nightly build that last passed successfully,  and deploys it to a machine somewhere. It does not compile, or run any tests**. It publishes the Installer it used.** takes 30 seconds. - The Deploy (to Staging) will just get the latest successful installer build artifacts from (deploy to test) builds, and deploy them to staging, and also **publish** the installer it used. Takes 30 seconds. - The Deploy (to Production) will just get the latest successful installer build artifacts from (deploy to staging) builds, and deploy them to production, and also **publish** the installer it used. Takes 30 seconds. Notice how with artifact reuse, I am able to reverse the time trend with builds. The more “advanced” a build is along the deployment pipeline, the faster it can become. And Because each build in the deploy pipeline is only getting artifacts of successful builds, we can be sure that if we got all the way to this stage, then all needed steps have been taken to be able to arrive at this situation (we ran all the tests, compiled all the source…) --- title: "Pattern: Script Injection" url: "https://pipelinedriven.org/article/2014/3/2/pattern-script-injection.md" canonical: "https://pipelinedriven.org/article/2014/3/2/pattern-script-injection" published: "2014-03-02T21:22:42.865000+00:00" updated: "2014-03-02T21:22:58.673000+00:00" tags: [] author: "Roy Osherove" description: "I am slowly realizing that perhaps my concepts for the build book could be better serving as reusable patterns for creating builds, solving specific…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Pattern: Script Injection 2 March 2014 · 2 min read · Roy Osherove I am slowly realizing that perhaps my concepts for [the build book](http://beautifulbuilds.com/) could be better serving as reusable patterns for creating builds, solving specific problems. This is a test to see how well this idea holds up in reality. If you like it, and especially if you do NOT like it, please let me know in the comments why, and how you would change it. I think this is a topic that can be of great use to many people. If we frame it right, it can be more easily distributed and understood. I am not even sure if the sections below are what I would “need” for a pattern of sorts. ## Pattern: Script Injection ## Problem: You have a continuous integration process running nicely, but sometimes you need to be able to build older versions of your product. Unfortunately, in older versions of your product in source control, the structure of files is different than the one that your build actions are set up to use. For example, a set of directories that exists in the latest version in source control, and is used in deployment, does not exist in the source control version from 3 months ago. So your build fails because it expects certain files or directories to be there, which did not exist in that source control version. ## Forces: You want the set of automated actions in your build to match exactly the current version and structure of your product files in source control. ## Solution: Separate your build script actions into two parts: - The CI side script, which lives in the continuous integration system. - The source control side script. ## Source control side script one or more script files that are inside the file structure of your product in source control. These scripts change based on the current product version. It is important that this is the same branch that is used within the CI system to build the product, so that the CI side scripts have access to the source control side scripts. Developers should have full access to the source control side scripts. Source Control side scripts contain the knowledge about the current structure of the files, and which actions are relevant for the current product version. So they get to be changed with every product version. They will usually use relative path because they will be executed by CI side scripts on a remote build server. ## CI Side Scripts/Actions These actions act as very simple “dumb” agents. They get the latest version of the source control scripts (and possibly all other product files if needed), and trigger the build scripts as a command line action. ## Summary By separating the version aware knowledge to a script inside source control, and triggering it via a CI side script that contains only parameters and other “context” data to invoke the source control scripts correctly, we “inject” the version aware knowledge of what the build script should do, into a higher level CI process trigger, that does not care about product file structure, but still gives us the advantages with had before with a CI process. **Thoughts? Comments? Does this make sense to you? is it the stupidest thing ever?** Does the fact that [I now wear glasses](https://twitter.com/RoyOsherove/status/285360033015947264/photo/1) help in any way? --- title: "Pattern: Shipping Skeleton" url: "https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton.md" canonical: "https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton" published: "2014-03-02T21:21:40.458000+00:00" updated: "2014-03-02T21:21:58.062000+00:00" tags: [] author: "Roy Osherove" description: "I am slowly realizing that perhaps my concepts for the build book could be better serving as reusable patterns for creating builds, solving specific…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Pattern: Shipping Skeleton 2 March 2014 · 2 min read · Roy Osherove I am slowly realizing that perhaps my concepts for [the build book](http://beautifulbuilds.com/) could be better serving as reusable patterns for creating builds, solving specific problems. ## Pattern: Shipping Skeleton other names: “Hello World”, Walking Skeleton ([based on XP](http://alistair.cockburn.us/Walking+skeleton)), Tracer Bullet Build, ## Problem: Remember when you had a working product, but you could not ship it, or shipping and deploying it took a long time, or could not be estimated?. By that time, however, your product was too big and your free time was too short to start creating a working automated build and deploy process. As time went on, shipping became even more and more of a nightmarish manual task, and automating it became more and more of a problem. Now you're at the start of a new project, and you want to avoid all that pain in your new project. ## Forces: You want to avoid the pain of automating the build and deploy cycle. But you also don’t want to spend too much time working on it. ## Solution: Before starting development on the new product, **start with a shipping skeleton --** An empty solution with nothing but a “hello world” output, that has the basic automated build and deploy cycle working. You should be able to ship this empty hello world project with the click of a button, in a matter of minutes. once the shipping skeleton is in place, you can start filling out the product with features, and growing the build scripts in small increments alongside the product. Basic Shipping Skeleton: 1) A Build script (in source control) for compiling the current source 2) A Continuous Integration Server that has a “CI” build configuration, triggered by code check-in, that invokes build script from the previous bullet (also see [Build Script injection](https://pipelinedriven.org/article/2014/3/2/pattern-script-injection) for more on this pattern) 3) Another build configuration on the CI server for “**Deploy**”. This can be either deploy to test, or deploy to production. Usually you want at least a “deploy to test” succeeding before having a “deploy to production” CI build Configuration. This “Deploy to test” gets invoked automatically upon “CI” configuration passing. Later on, as the build process matures, you can change this, but for now, knowing that the product is deployable as soon as possible, while it changes so much, is important for quick feedback cycles. 4) If there isn’t a “Deploy to production”, create it. This deploys the product to a production machine. If it is a web site, it deploys the website to a web server. if it is an installer, it deploys and runs the installer on a machine that will act as a “production” machine. either mimicking a user machine, or or an enterprise machine where the product will be installed. for web servers, make them as real as possible, all the way, even by making them public (although possibly password protected). This gives you the chance to make the web server become real production by the switch of a dns setting. ## Summary By starting with a shipping skeleton, you give yourself several benefits: - You can ship at will - Adding features to the build and deploy cycle is a continuous action that takes a few minutes each day, if at all. - You can receive very quick feedback on features or mockup-features you are building into your product. **Thoughts? Comments? Does this make sense to you? is it the stupidest thing ever?** --- title: "Build Pattern: Location Agnostic Script" url: "https://pipelinedriven.org/article/2014/3/2/build-pattern-location-agnostic-script.md" canonical: "https://pipelinedriven.org/article/2014/3/2/build-pattern-location-agnostic-script" published: "2014-03-02T21:20:02.941000+00:00" updated: "2014-03-02T21:20:43.890000+00:00" tags: [] author: "Roy Osherove" description: "Symptoms: Running the build script on a machine where it was never run before requires extra pre-work to make the machine ready, such as mapping drives,…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Build Pattern: Location Agnostic Script 2 March 2014 · 1 min read · Roy Osherove ## Symptoms: - Running the build script on a machine where it was never run before requires extra pre-work to make the machine ready, such as mapping drives, creating a special build script folder etc. ## Problem: - The build script uses hardcoded file and directory paths to find its dependent source files. For example, it searches for a solution file in Z:/SolutionFileName . That means the build script has specific requirements from its environment before running, which causes lots of menial, boring work to do before being able to run it. ## Solution: - Have the script use only relative paths to find files. - Make the script part of source control, or deploy it into the root of the source control branch to build, so that it has access to all the files it needs --- title: "Release Compilation" url: "https://pipelinedriven.org/article/2014/3/2/release-compilation.md" canonical: "https://pipelinedriven.org/article/2014/3/2/release-compilation" published: "2014-03-02T21:04:31.743000+00:00" updated: "2014-03-02T21:07:38.564000+00:00" tags: [] author: "Roy Osherove" description: "Imagine you could only compile once a week. Once every two weeks. Once a month. Scary? All that code that you write blindly, not knowing whether it even…" usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Release Compilation 2 March 2014 · 6 min read · Roy Osherove ## Imagine you could only compile once a week. Once every two weeks. Once a month. Scary? All that code that you write blindly, not knowing whether it even compiles. All those hidden errors lurking, accumulating in the dark. Then, on that special day of compilation, you get to finally find out if you made it. And of course, you most likely did not. Unfortunately, that special compilation day is followed by an even worse day. Demo day. You are supposed to sit down with your customer, or product owner, and show them all the progress you have made to your application. Most times, compilation day goes sour, and you are stuck fixing last minute, last hour, and last night-Compilation issues only to find out that now, the application isn’t working. So there goes demo day. You secretly start wondering if there was a way to ‘break the rules’. A way that you could secretly compile your code — maybe on a daily basis?— and see if you left something out. Hell, why not do it every time you check in your code? or even save it? What if you could \_continuously\_ compile your code? Your life would be so much easier. Not to mention your customer’s life. You could find those compilation errors quickly, and thus fix them quickly while they are still fresh in memory. You could start focusing on that demo now that the code compiles, and see if the app is actually working! Aren’t we lucky that we all already live with that kind of continuous compilation? At least in static languages, we can see if our code ‘checks out’ by the compiler almost instantly. We can write a single line, or even a single code keyword, and see if it works when the reality of the compiler hits. If we are working in a dynamically typed language, like Ruby or Python, we write tests, and we run those tests continuously , so that we get feedback that the code works. Well, some of us do, anyway. A lot of us just \_say\_ we do it, but deep down we know it is lip service. Sometimes we get a ‘fluke’ - we write a function and it fully compiles and runs with no problems with no compilation and debugging errors. That kind of magic doesn’t happen very often though. Things aren’t perfect. Now that we live in this magical world where compilation is continuous, we start to realize we still have problems with our customer’s demos. The demos aren’t working. Well, they _are_ working, but not really. they break and they work only partially. And when we try to fix the functionality problems, we break other functionality. We need testing, or, more precisely, \_automated\_ testing to tell us if our code functions right. Why does it need to be automated though? Because manual testing takes so much time. It’s really a good waste of money to repeat the same regression tests by a human. We should be using humans for tasks with creativity built in, like exploration testing, but we need ‘automated checks’ for regression testing, to keep telling us we didn’t break anything, continuously. So we write some tests, and we run them locally every time we are about to check in the code. things feel better for a while, but then we realize the customer demos are still not really working. “Why would the customer tests not be working \_now\_?” you ask yourself. Turns out, the demos are supposed to be shown on the customer’s machine. The customer’s machine is very different from a developer’s machine. It has weird firewall requirements, it needs active directory permissions to run your application, and the database is shared on a different machine. For you, demo day is also, it turns out, deployment day. Because you deploy before you demo (usually the day before if you are lucky), it means you have one day every week you deploy. Or maybe demo is once every two or three weeks, so you deploy just once every two or three weeks. If you’re lucky. Deployment once a week? Now you spend your week compiling and testing code, but you’re still as blind as a bat. You do not know whether all this code will actually work within the constraints of the customer’s machine. Sure, it works on \_your machine\_, but we all know that doesn’t mean much.You spend all this time writing and polishing your beautiful code, all the while you have no idea whether it will \_fit in\_ with the hard reality of living on a customer’s machine. Demo day comes, and you deploy, and your perfectly working code is shamingly crashing, lagging, stalling and basically acting like an angry child in the middle of a supermarket: Badly. Your demo sucks. And now you can’t get any feedback from the customer. Instead, you have wasted your customer’s time, and some of their hard earned trust. This keeps happening. Some weeks you get things right, some weeks the new pieces of code just don’t fit the configuration. You have to wait a week between demos to find out if the changes you made to your code or configuration files actually work, because, well, _you deploy once a week_. You secretly start wondering what happened if you broke the rules and got your own little machine that looks and acts just like a customer’s machine. If you had that kind of machine locally, you could try and deploy to it on a daily basis, and find out if your code actually works after deployment. If it didn’t you could fix it way before demo day. And then you can use the demos to actually get feedback on the functionality of your application. On second thought, what if you went all out and \_automated\_ the deployment to the fake customer machine? then you could test your deployment continuously! On every check in. Now that would be quite beautiful. Deployment testing is like a compilation step for your release cycle. It verifies that your product, when deployed, can face the harsh realities of a production machine. It should happen continuously, so that you do not find out at the last minute that your code or configuration, is not able to handle the current reality. For deployment testing, you need to create a \_staging\_ environment. This environment mimics the production environment as closely as possible, and it should not be contaminated with compilers, editors or development tools. How do you test your deployment with a staging environment? You run acceptance tests ,and hopefully also all the other automated tests you’ve written, against the application deployed on staging. For example, you could run browser tests (using selenium or capybara — google those) against a web application installed on staging. What happens if you \_do\_ need to debug something that only seems to happen in the staging environment? That’s why you have a ‘test’ environment. think of ‘test’ as ‘staging + debugging tools’. It can be ‘dirty’ in that you can use it to examine things more closely, but in an environment that can simulate real world difficulties. The environment right before ‘test’ is ‘dev-test’. that can either be a developer local machine, or a continuous integration build agent, that usually just compiles and runs automated tests. To me, a beautiful build, is one that encompasses all the levels of confidence we just discussed: Compilation, automated testing, automated deployment, and deployment testing. If possible,I take the next logical step - I also deploy to a production machine in an automated manner - the same code that has passed all the other stages. I, and many others, call this chain ‘Continuous Delivery’. (will be part of my builds book) --- title: "Pipeline Driven: Continuous delivery, build patterns and enterprise DevOps" url: "https://pipelinedriven.org/index.md" canonical: "https://pipelinedriven.org/" tags: [] author: "Roy Osherove" description: "Pipeline Driven by Roy Osherove: continuous delivery and DevOps best practices, build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Pipeline Driven > Pipeline Driven is Roy Osherove's site about continuous delivery: build pipelines, pipeline-friendly testing, DevOps metrics and enterprise DevOps. HTML: https://pipelinedriven.org/ · Guide for AI systems: [llms.txt](https://pipelinedriven.org/llms.txt) · Full text: [llms-full.txt](https://pipelinedriven.org/llms-full.txt) ## On the home page Pipeline Driven · by Roy Osherove ### What is a pipeline driven organization? A Pipeline Driven Organization is operating around the idea that **automated pipelines make the important IT decisions**: 1. Commit - Is the code releasable? 2. Build - Are the infrastructure changes OK? - Are we compliant and secure? 3. Test - Is a feature working? 4. Deploy - When is it OK to deploy a new version? - When and how to deploy a new environment 5. Learn - When to rollback (or forward) a new version [Read the blog](https://pipelinedriven.org/article) [Watch the Øredev 2019 talk](https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019) #### Who makes the call? Traditional organizations In traditional organizations, even though automated pipelines may exist, they do not make the important & continuous IT decisions. **Humans do.** Pipeline driven organizations In a pipeline based organization, experts from various categories of work **spend their time teaching the pipelines** how to make proper decisions (using automated specialized tests). In this way they can trust the pipeline enough to let it make the decisions instead. Diagram #### Co-Ops - A Cooperative Pipeline [![Co-Ops - A Cooperative Pipeline: a diagram of the delivery pipeline with the expert groups that feed decisions into it](https://pipelinedriven.org/img/c/screen-shot-2019-11-21-at-12.47.58-am-0109cf.png)](https://pipelinedriven.org/img/c/screen-shot-2019-11-21-at-12.47.58-am-0109cf.png) Co-Ops - A Cooperative Pipeline. [Read “Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)”](https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed) #### Latest posts Archive · 2014–2022 [More Posts](https://pipelinedriven.org/article) 1. [Metrics](https://pipelinedriven.org/article/line/metrics) 11 July 2022 ### [A Metrics Framework for Continuous Delivery](https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery) Here’s a framework I like to start with when I discuss what types of metrics can help or harm you in your journey to continuous delivery: Lagging Indicators At the organizational level, unit tests are usually part of a… 2. [Talks & Events](https://pipelinedriven.org/article/line/talks) 11 July 2022 Video ### [Video: Lies, Damned Lies and Metrics](https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics) I spoke about what types of metrics and KPIs (including OKRs) you can use with continuous delivery, and which ones you might want to stop using, during my talk at GOTO 2019. Also discussed: Leading and Lagging… 3. [Build Patterns](https://pipelinedriven.org/article/line/build) 19 April 2020 ### [If a Build Takes 4 hours, Run It Every 4 Hours](https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours) (Note: You might need to fix your environment bottlenecks first before being able to act on this blog post) Builds (i.e. compile + run tests + deploys + more tests etc…) represent a bottleneck for getting feedback on our… 4. [Enterprise DevOps](https://pipelinedriven.org/article/line/ent) 6 December 2019 ### [Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)](https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed) “Co-Ops” is a term that one of the attendees at a talk I was giving about cooperative pipelines came up with. I wish I knew his name. I asked him to email me afterwards so I can credit him but he never did. Co-Ops… 5. [Talks & Events](https://pipelinedriven.org/article/line/talks) 20 November 2019 Video ### [Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019)](https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019) Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube 6. [Talks & Events](https://pipelinedriven.org/article/line/talks) 2 June 2019 ### [Continuous Delivery Israel Meetup - First Meeting June 24 2019](https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019) I’m pleased to announce that we will have the first ever meeting Continuous Delivery Israel meetup, in the AWS Floor28 offices in Tel Aviv, June 24th, 2019, at 6PM. The Group’s page is located at CDIsrael.com If you… #### Browse by line Every post sits on one of six topic lines. - ### Build Patterns Fast, maintainable builds: shipping skeletons, script injection, rolling builds. - [If a Build Takes 4 hours, Run It Every 4 Hours](https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours) - [Guidelines for selecting tools for continuous delivery toolchains](https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains) - [The first rule of continuous delivery toolchains](https://pipelinedriven.org/article/the-first-rule-of-continuous-delivery-toochains) [All 17 posts on Build Patterns](https://pipelinedriven.org/article/line/build) - ### Testing Pipeline-friendly test strategies and trustworthy, noise-free pipelines. - [Delivery Pipelines and Discovery Pipelines](https://pipelinedriven.org/article/delivery-pipelines-and-discovery-pipelines) - [Red Pipelines and Build Whisperers: Getting to a Trustworthy Noise Free Test Pipeline](https://pipelinedriven.org/article/red-builds-and-build-whisperers-trustworthy-noise-free-test-pipelines) - [A Pipeline Friendly Layered Testing Strategy & Recipe for DEV and QA](https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa) [All 3 posts on Testing](https://pipelinedriven.org/article/line/test) - ### Metrics What to measure on the way to continuous delivery, and what not to. - [A Metrics Framework for Continuous Delivery](https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery) - [Ten Devops & Agility Metrics to Check at the Team Level](https://pipelinedriven.org/article/ten-ideas-for-things-you-can-measure-as-a-team-on-your-devops-journey) - [DevOps Metric: Amount of Defects and Issues: definition.](https://pipelinedriven.org/article/devops-metric-amount-of-defects-and-issues-definition) [All 7 posts on Metrics](https://pipelinedriven.org/article/line/metrics) - ### Environments Ephemeral environments and the cost of static ones. - [Ephemeral Environments for DevOps: Why, What and How?](https://pipelinedriven.org/article/ephemeral-environment-why-what-how-and-where) - [The four biggest issues with having static environments](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments) - [Parallel Firehose](https://pipelinedriven.org/article/2014/9/22/parallel-firehose) [All 3 posts on Environments](https://pipelinedriven.org/article/line/env) - ### Enterprise DevOps DevOps values at scale: approvals, monoliths, compliance, culture. - [Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)](https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed) - ["Pipeline Driven" vs. "We have Jenkins Jobs"](https://pipelinedriven.org/article/pipeline-driven-vs-we-have-jenkins-jobs) - [Always start with a "Why" when transforming to DevOps.](https://pipelinedriven.org/article/always-start-with-a-why-when-transforming-to-devops) [All 7 posts on Enterprise DevOps](https://pipelinedriven.org/article/line/ent) - ### Talks & Events Conference talks, interviews and meetups. - [Video: Lies, Damned Lies and Metrics](https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics) - [Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019)](https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019) - [Continuous Delivery Israel Meetup - First Meeting June 24 2019](https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019) [All 4 posts on Talks & Events](https://pipelinedriven.org/article/line/talks) #### Resources - [WatchTalks and videosConference talks on pipelines, enterprise DevOps and testing.](https://pipelinedriven.org/videos) - [LibraryBeautiful Builds8 short videos and the book on build patterns.](https://pipelinedriven.org/library) - [ReadRecommended DevOps booksRoy’s shortlist, from Continuous Delivery to The Goal.](https://pipelinedriven.org/enterprise-devops-books) - [ToolsFeature toggle frameworksFeature flag libraries by language.](https://pipelinedriven.org/feature-toggle-frameworks-list) ![Roy Osherove](https://pipelinedriven.org/img/author/roy-osherove-2026-face-480.jpg) #### By Roy Osherove Roy Osherove helps software organizations deliver continuously. He is the author of [The Art of Unit Testing](https://www.artofunittesting.com/), [Elastic Leadership](https://www.elasticleadership.com/) and [Enterprise DevOps](https://pipelinedriven.org/books-by-roy-osherove), and speaks about pipelines, testing and team leadership at conferences such as Øredev. [Get in touch](https://forms.gle/jQSx5KoXvrVA1JCK7) ## Pages - [What is Enterprise DevOps?](https://pipelinedriven.org/enterprise-devops.md): What is Enterprise DevOps? The application of DevOps values in organizations with interdependent systems, monoliths, approval gates and compliance requirements. - [Videos](https://pipelinedriven.org/videos.md): Conference talks by Roy Osherove on the pipeline driven organization, enterprise DevOps and testing, plus the Beautiful Builds video library. - [Feature Toggles](https://pipelinedriven.org/feature-toggle-frameworks-list.md): A list of feature toggle (feature flag, feature bit) frameworks for Java, Python, .NET, Ruby and more. - [Books by Roy Osherove](https://pipelinedriven.org/books-by-roy-osherove.md): Books by Roy Osherove: The Art of Unit Testing, Elastic Leadership and Enterprise DevOps. - [Enterprise DevOps Books](https://pipelinedriven.org/enterprise-devops-books.md): Roy Osherove's favourite books about DevOps, enterprise DevOps, continuous delivery and lean. - [Beautiful Builds library](https://pipelinedriven.org/library.md): The Beautiful Builds library: short videos by Roy Osherove on build patterns, continuous integration and deployment automation. - [DevOps & Enterprise DevOps Recommended Books](https://pipelinedriven.org/enterprise-devops-books-list.md): Pipeline Driven by Roy Osherove: continuous delivery and DevOps best practices, build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Books by Roy Osherove](https://pipelinedriven.org/agility-devops-related-books.md): Pipeline Driven by Roy Osherove: continuous delivery and DevOps best practices, build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Contact](https://pipelinedriven.org/contact.md): Contact Roy Osherove about continuous delivery, enterprise DevOps, consulting or training. ## Builds - [Faster Feedback, Waves of Confidence, and Automatic Triggers](https://pipelinedriven.org/library/2014/2/18/faster-feedback-waves-of-confidence-and-automatic-triggers.md): Faster Feedback, Waves of Confidence, and Automatic Triggers: from the Beautiful Builds library by Roy Osherove. - [Start by Shipping : Shipping Skeleton](https://pipelinedriven.org/library/2014/2/18/start-by-shipping-shipping-skeleton.md): Start by Shipping : Shipping Skeleton: from the Beautiful Builds library by Roy Osherove. - [The Beautiful Builds Book](https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book.md): The Beautiful Builds Book: from the Beautiful Builds library by Roy Osherove. - [The Big Difference Betweeen Automated Builds and Continuous Integration](https://pipelinedriven.org/library/2014/2/18/the-big-difference-betweeen-automated-builds-and-continuous-integration.md): The Big Difference Betweeen Automated Builds and Continuous Integration: from the Beautiful Builds library by Roy Osherove. - [Beautiful Builds - Roy Osherove](https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove.md): Beautiful Builds - Roy Osherove: from the Beautiful Builds library by Roy Osherove. - [App Deployment and Automation - 20 tools and tips by Roy Osherove](https://pipelinedriven.org/library/2014/2/18/app-deployment-and-automation-20-tools-and-tips-by-roy-osherove.md): App Deployment and Automation - 20 tools and tips by Roy Osherove: from the Beautiful Builds library by Roy Osherove. - [API Artifacts and Inter-Component Dependencies](https://pipelinedriven.org/library/2014/3/2/api-artifacts-and-inter-component-dependencies.md): API Artifacts and Inter-Component Dependencies: from the Beautiful Builds library by Roy Osherove. - [Introduction to Beautiful Builds](https://pipelinedriven.org/library/2014/3/2/introduction-to-beautiful-builds.md): Introduction to Beautiful Builds: from the Beautiful Builds library by Roy Osherove. ## DevOps & Enterprise DevOps Recommended Books - [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6.md): Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation: a book recommended on Pipeline Driven by Roy Osherove. - [The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5.md): The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations: a book recommended on Pipeline Driven by Roy Osherove. - [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6.md): Enterprise DevOps: Patterns for Succeeding with DevOps at Scale: a book recommended on Pipeline Driven by Roy Osherove. - [Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu.md): Continuous Integration: Improving Software Quality and Reducing Risk: a book recommended on Pipeline Driven by Roy Osherove. - [The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc.md): The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014: a book recommended on Pipeline Driven by Roy Osherove. - [The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1.md): The Goal: A Process of Ongoing Improvemen: a book recommended on Pipeline Driven by Roy Osherove. ## Books by Roy Osherove - [The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing.md): The Art of Unit Testing, 2nd Edition: a book recommended on Pipeline Driven by Roy Osherove. - [Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership.md): Elastic Leadership: Growing Self Organizing Teams: a book recommended on Pipeline Driven by Roy Osherove. - [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops.md): Enterprise DevOps: Patterns for Succeeding with DevOps at Scale: a book recommended on Pipeline Driven by Roy Osherove. ## Posts (41) ### Build Patterns - [If a Build Takes 4 hours, Run It Every 4 Hours](https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours.md) (2020-04-19): (Note: You might need to fix your environment bottlenecks first before being able to act on this blog post) Builds (i.e. compile + run tests + deploys +… - [Guidelines for selecting tools for continuous delivery toolchains](https://pipelinedriven.org/article/the-rules-of-selecting-tools-for-continuous-delivery-toolchains.md) (2017-03-10): If the tool you use does not support continuous delivery values , you're going to have a bad time implementing CI/CD with fully automated pipelines. Here… - [The first rule of continuous delivery toolchains](https://pipelinedriven.org/article/the-first-rule-of-continuous-delivery-toochains.md) (2017-03-10): Continuous delivery transformations are hard enough in many areas already: changing people's behavior, changing policies, architecting things differently -… - [BBBCTC - Branching, Binaries, Builds, Coding and Testing Culture](https://pipelinedriven.org/article/2015/5/30/bbbctc-branching-binaries-builds-coding-and-testing-culture.md) (2015-05-30): As part of a major drive to push for faster "time to main" you might need to change things in the following five areas (that end up affecting how long… - [Pipeline Disintegration (Post Build Pipeline)](https://pipelinedriven.org/article/2015/3/29/pipeline-disintegration-post-build-pipeline.md) (2015-03-29): Other Names: Post Build Hand off, Time Capsule, Strangler Application Symptoms: It is very complicated to add features to the current build process, because… - [Split and Parallelize](https://pipelinedriven.org/article/2014/10/3/q5b4rm6abtjxeleaj0xfapo1n3khnl.md) (2014-10-03): Other Names: Split Work Symptoms: The build is taking too long Problem: It could be that one or more tasks or build steps that are comprised of many actions… - [Irrelevant Build](https://pipelinedriven.org/article/2014/10/3/-irrelevant-build.md) (2014-10-03): Symptoms: The team doesn't seem to care that one of the builds is failing A build is red, but might not be failing. Only one person from a different team or… - [Build Pattern: Deploy by Proxy - Added to the book](https://pipelinedriven.org/article/2014/7/26/build-pattern-deploy-by-proxy-added-to-the-book.md) (2014-07-26): It's been a couple months of other work. but there's finally new material in the book and more coming soon. First: I've now grouped the chapters under… - [Use Version Aware Build Scripts](https://pipelinedriven.org/article/2014/3/2/use-version-aware-build-scripts.md) (2014-03-02): I am thinking of writing a small booklet about better builds . this blog post might end up as a part of it . Let me know if you’re interested by taking a… - [Avoid XML Facing Build Tools](https://pipelinedriven.org/article/2014/3/2/avoid-xml-facing-build-tools.md) (2014-03-02): Having your build scripts in XML is one of the worst things you can do for maintainability and readability. Treat your build scripts like you treat your… - [Start Products with an Empty Build and Deploy Pipeline](https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline.md) (2014-03-02): Builds can be nasty beasts when It’s finally time to organize and do them. There are so many things you did not think about, that it can take days or weeks… - [Rolling Builds and the Plane of Confidence](https://pipelinedriven.org/article/2014/3/2/rolling-builds-and-the-plane-of-confidence.md) (2014-03-02): When I check in code I have been working on, I feel: 1. I want to wait for the build result 2. I don’t want to waste time waiting for the build results,… - [How to make your builds FAST using DRY and SRP Principles](https://pipelinedriven.org/article/2014/3/2/how-to-make-your-builds-fast-using-dry-and-srp-principles.md) (2014-03-02): (this will become part of my upcoming booklet on beautiful builds ) One of the things that kept frustrating me when I was working on various types of build… - [Pattern: Script Injection](https://pipelinedriven.org/article/2014/3/2/pattern-script-injection.md) (2014-03-02): I am slowly realizing that perhaps my concepts for the build book could be better serving as reusable patterns for creating builds, solving specific… - [Pattern: Shipping Skeleton](https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton.md) (2014-03-02): I am slowly realizing that perhaps my concepts for the build book could be better serving as reusable patterns for creating builds, solving specific… - [Build Pattern: Location Agnostic Script](https://pipelinedriven.org/article/2014/3/2/build-pattern-location-agnostic-script.md) (2014-03-02): Symptoms: Running the build script on a machine where it was never run before requires extra pre-work to make the machine ready, such as mapping drives,… - [Release Compilation](https://pipelinedriven.org/article/2014/3/2/release-compilation.md) (2014-03-02): Imagine you could only compile once a week. Once every two weeks. Once a month. Scary? All that code that you write blindly, not knowing whether it even… ### Enterprise DevOps - [Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)](https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed.md) (2019-12-06): “Co-Ops” is a term that one of the attendees at a talk I was giving about cooperative pipelines came up with. I wish I knew his name. I asked him to email… - ["Pipeline Driven" vs. "We have Jenkins Jobs"](https://pipelinedriven.org/article/pipeline-driven-vs-we-have-jenkins-jobs.md) (2019-05-03): High performing and continuously delivering organizations such as Netflix, Amazon, Google, Wix and others don’t just have pipelines. They don’t just… - [Always start with a "Why" when transforming to DevOps.](https://pipelinedriven.org/article/always-start-with-a-why-when-transforming-to-devops.md) (2018-10-21): I’ve been to quite a few DevOps and Agile transformations, big and small. Some just want to up the coverage of their tests (not recommended!), others want… - [Continuous Delivery Values](https://pipelinedriven.org/article/continuous-delivery-values.md) (2017-03-10): I mentioned in my last post about the toolchain needing to respect the continuous delivery values . What are those values? Many of them derive from lean… - [The four most common root causes that slow down enterprise continuous delivery](https://pipelinedriven.org/article/the-four-most-common-root-causes-that-slow-down-enterprise-continuous-delivery.md) (2017-03-09): In my journey as a developer, architect, team leader, CTO, director, coach and consultant for software development, the two most common "anti patterns" I… - [What is Enterprise DevOps?](https://pipelinedriven.org/article/what-is-enterprise-devops.md) (2016-10-29): To understand Enterprise DevOps, we have to first define what "regular" DevOps is. Here is my current definition of DevOps. Let's call it "DevOps 0.1" What… - [Devops is...](https://pipelinedriven.org/article/2015/10/29/devops-is.md) (2015-10-29): Infrastructure & Code Treated as software, where Software Is created with fast feedback cycles, where Cycles Do not require monkey work by people, where… ### Metrics - [A Metrics Framework for Continuous Delivery](https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery.md) (2022-07-11): Here’s a framework I like to start with when I discuss what types of metrics can help or harm you in your journey to continuous delivery: Lagging Indicators… - [Ten Devops & Agility Metrics to Check at the Team Level](https://pipelinedriven.org/article/ten-ideas-for-things-you-can-measure-as-a-team-on-your-devops-journey.md) (2018-03-10): When I coach teams that are getting into the DevOps and Continuous Delivery mindset, a common question that comes up is "What should we measure?" Measuring… - [DevOps Metric: Amount of Defects and Issues: definition.](https://pipelinedriven.org/article/devops-metric-amount-of-defects-and-issues-definition.md) (2016-10-29): Important to : VP of QA, Operations Definition : How many bugs or tickets are found in production/ how many recurring bugs in production. As well as - how… - [DevOps Metric: Frequency of Releases Definition](https://pipelinedriven.org/article/devops-metric-frequency-of-releases-definition.md) (2016-10-29): Important to : Release Manager, CIO Definition : How often an official release is deployed to production, in paying customer's hands How to measure : If no… - [DevOps Metric: Mean time to recovery (MTTR) Definition and reasoning](https://pipelinedriven.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning.md) (2016-10-29): Important to: VP Operations, CTO Definition :When a production failure occurs, how long does it take to recover from the issue? How to measure: Different… - [DevOps Metric Definition: Mean time to change (MTTC) (vs change lead time)](https://pipelinedriven.org/article/devops-metric-definition-mean-time-to-change-mttc-vs-change-lead-time.md) (2016-10-29): Also called "time from concept to cash". Important to : Everyone, especially the CEO, CIO and CTO. Definition : How long does it take a new average feature,… - [Time to Main - Measuring Continuous Integration](https://pipelinedriven.org/article/2015/5/15/time-to-main-measuring-continuous-integration.md) (2015-05-15): A.k.a "Time to Trunk". How do you measure your progress on the way to continuous integration(we're not even discussing delivery yet)? How do you know you… ### Talks & Events - [Video: Lies, Damned Lies and Metrics](https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics.md) (2022-07-11): I spoke about what types of metrics and KPIs (including OKRs) you can use with continuous delivery, and which ones you might want to stop using, during my… - [Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019)](https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019.md) (2019-11-20): Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube - [Continuous Delivery Israel Meetup - First Meeting June 24 2019](https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019.md) (2019-06-02): I’m pleased to announce that we will have the first ever meeting Continuous Delivery Israel meetup, in the AWS Floor28 offices in Tel Aviv, June 24th, 2019,… - [Interview on Devops and Cloud Infrastructure](https://pipelinedriven.org/article/2016/1/7/interview-on-devops-and-cloud-infrastructure.md) (2016-01-07): I've had the pleasure of being part of a panel on Devops and various patterns and anti patterns relating to cloud operations and continuous delivery. You… ### Environments - [Ephemeral Environments for DevOps: Why, What and How?](https://pipelinedriven.org/article/ephemeral-environment-why-what-how-and-where.md) (2017-03-30): In the previous post I discussed the issues with having static environments . In this post I cover one of the solutions for those issues: Ephemeral… - [The four biggest issues with having static environments](https://pipelinedriven.org/article/the-four-biggest-issues-with-having-static-environments.md) (2017-03-30): An environment is a set of one or more servers, configured to host the application we are developing, and with the application already installed on them,… - [Parallel Firehose](https://pipelinedriven.org/article/2014/9/22/parallel-firehose.md) (2014-09-22): Pattern: Parallel Firehose Other Names: Build Environment Per Stakeholder Symptoms: You can't deploy a new version of the application to your test or… ### Testing - [Delivery Pipelines and Discovery Pipelines](https://pipelinedriven.org/article/delivery-pipelines-and-discovery-pipelines.md) (2019-02-20): In a previous post I discussed the notion of having two seperate pipelines. I want to revisit this, with a bit more details. When we’re talking about… - [Red Pipelines and Build Whisperers: Getting to a Trustworthy Noise Free Test Pipeline](https://pipelinedriven.org/article/red-builds-and-build-whisperers-trustworthy-noise-free-test-pipelines.md) (2019-02-18): People wonder why it takes such a long time to transform the way a large organization works. Here’s just a small nugget, a small tip of a small iceberg, in… - [A Pipeline Friendly Layered Testing Strategy & Recipe for DEV and QA](https://pipelinedriven.org/article/a-pipeline-friendly-layered-testing-strategy-amp-recipe-for-dev-and-qa.md) (2019-02-09): The key point of running a pipeline is to get feedback which in turn is supposed to provide us with Confidence. That’s because a pipeline is really just a… --- title: "What is Enterprise DevOps?" url: "https://pipelinedriven.org/enterprise-devops.md" canonical: "https://pipelinedriven.org/enterprise-devops" tags: [] author: "Roy Osherove" description: "What is Enterprise DevOps? The application of DevOps values in organizations with interdependent systems, monoliths, approval gates and compliance requirements." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # What is Enterprise DevOps? Enterprise DevOps [**Enterprise DevOps**](https://pipelinedriven.org/article/what-is-enterprise-devops) is the application of DevOps values in an environment that contains any of the following: - Many **inter-dependent** and related systems and sub systems, software and teams that rely on each other - **Monolithic** systems and/or static software/hardware environments - **Lengthy** Approval Gates or change control processes - **Security**, **financial**, **medical** or other **compliance** requirements - Lengthy **waterfall** based processes among multiple teams and stake holders - Workflows that are mostly **manual** and **error prone** across multiple teams or stakeholders In order to create an organization that operates through automated pipelines of continuous delivery. [![](https://pipelinedriven.org/img/yt/K6ZQxh5hzWg.jpg)](https://www.youtube.com/watch?v=K6ZQxh5hzWg) Roy Osherove - Patterns & Anti Patterns for Scaling & Enabling DevOps Adoption · YouTube [Learn More](https://pipelinedriven.org/article/what-is-enterprise-devops) ## Latest posts [More posts →](https://pipelinedriven.org/article) 1. [Metrics](https://pipelinedriven.org/article/line/metrics) 11 July 2022 ### [A Metrics Framework for Continuous Delivery](https://pipelinedriven.org/article/a-metrics-framework-for-continuous-delivery) Here’s a framework I like to start with when I discuss what types of metrics can help or harm you in your journey to continuous delivery: Lagging Indicators At the organizational level, unit tests are usually part of a… 2. [Talks & Events](https://pipelinedriven.org/article/line/talks) 11 July 2022 Video ### [Video: Lies, Damned Lies and Metrics](https://pipelinedriven.org/article/video-lies-damned-lies-and-metrics) I spoke about what types of metrics and KPIs (including OKRs) you can use with continuous delivery, and which ones you might want to stop using, during my talk at GOTO 2019. Also discussed: Leading and Lagging… 3. [Build Patterns](https://pipelinedriven.org/article/line/build) 19 April 2020 ### [If a Build Takes 4 hours, Run It Every 4 Hours](https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours) (Note: You might need to fix your environment bottlenecks first before being able to act on this blog post) Builds (i.e. compile + run tests + deploys + more tests etc…) represent a bottleneck for getting feedback on our… 4. [Enterprise DevOps](https://pipelinedriven.org/article/line/ent) 6 December 2019 ### [Co-Ops: Enabling True Continuous Delivery with Cooperative Pipelines & Processes (Title Feedback Needed)](https://pipelinedriven.org/article/co-ops-enabling-true-continuous-delivery-with-cooperative-pipelines-amp-processes-title-feedback-needed) “Co-Ops” is a term that one of the attendees at a talk I was giving about cooperative pipelines came up with. I wish I knew his name. I asked him to email me afterwards so I can credit him but he never did. Co-Ops… 5. [Talks & Events](https://pipelinedriven.org/article/line/talks) 20 November 2019 Video ### [Video: The Pipeline Driven Organization - Enabling True Continuous Delivery (Oredev 2019)](https://pipelinedriven.org/article/video-the-pipeline-driven-organization-enabling-true-continuous-delivery-oredev-2019) Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube 6. [Talks & Events](https://pipelinedriven.org/article/line/talks) 2 June 2019 ### [Continuous Delivery Israel Meetup - First Meeting June 24 2019](https://pipelinedriven.org/article/continuous-delivery-israel-meetup-first-meeting-june-24-2019) I’m pleased to announce that we will have the first ever meeting Continuous Delivery Israel meetup, in the AWS Floor28 offices in Tel Aviv, June 24th, 2019, at 6PM. The Group’s page is located at CDIsrael.com If you… --- title: "Videos" url: "https://pipelinedriven.org/videos.md" canonical: "https://pipelinedriven.org/videos" tags: [] author: "Roy Osherove" description: "Conference talks by Roy Osherove on the pipeline driven organization, enterprise DevOps and testing, plus the Beautiful Builds video library." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Videos Watch Talks by Roy Osherove. Videos load from YouTube or Vimeo only when you press play. [![](https://pipelinedriven.org/img/yt/bz4AKIIqGL4.jpg)](https://www.youtube.com/watch?v=bz4AKIIqGL4) Roy Osherove — The Pipeline Driven Organization｜ Øredev 2019 · YouTube [![](https://pipelinedriven.org/img/yt/K6ZQxh5hzWg.jpg)](https://www.youtube.com/watch?v=K6ZQxh5hzWg) Roy Osherove - Patterns & Anti Patterns for Scaling & Enabling DevOps Adoption · YouTube [![](https://pipelinedriven.org/img/yt/YXuXarP43xM.jpg)](https://www.youtube.com/watch?v=YXuXarP43xM) Opening Keynote: Testing in the age of DevOps by Roy Osherove · YouTube [VimeoThe Enterprise DevOps Challenge - Roy OsheroveWatch on Vimeo ↗](https://vimeo.com/284918200) [![](https://pipelinedriven.org/img/yt/jji5ruv60mA.jpg)](https://www.youtube.com/watch?v=jji5ruv60mA) Opening Keynote: The Enterprise DevOps Mind Trap by Roy Osherove · YouTube [![](https://pipelinedriven.org/img/yt/jQ66f4MePNs.jpg)](https://www.youtube.com/watch?v=jQ66f4MePNs) Roy Osherove - Ten Challenges in Enterprise DevOps · YouTube ## Beautiful Builds library Short videos on build patterns, from the [Beautiful Builds](https://pipelinedriven.org/library) collection. - [![](https://pipelinedriven.org/img/yt/QuOOKrzrjfg.jpg)](https://www.youtube.com/watch?v=QuOOKrzrjfg) [Faster Feedback, Waves of Confidence, and Automatic Triggers](https://pipelinedriven.org/library/2014/2/18/faster-feedback-waves-of-confidence-and-automatic-triggers) - [![](https://pipelinedriven.org/img/yt/8jeIwGzuqJ8.jpg)](https://www.youtube.com/watch?v=8jeIwGzuqJ8) [Start by Shipping : Shipping Skeleton](https://pipelinedriven.org/library/2014/2/18/start-by-shipping-shipping-skeleton) - [![](https://pipelinedriven.org/img/c/title-page-244613.jpg)](https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book) [The Beautiful Builds Book](https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book) - [![](https://pipelinedriven.org/img/yt/mg5GzHbSZsw.jpg)](https://www.youtube.com/watch?v=mg5GzHbSZsw) [The Big Difference Betweeen Automated Builds and Continuous Integration](https://pipelinedriven.org/library/2014/2/18/the-big-difference-betweeen-automated-builds-and-continuous-integration) - [![](https://pipelinedriven.org/img/c/image-asset-3b82e3.jpg)](https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove) [Beautiful Builds - Roy Osherove](https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove) - [![](https://pipelinedriven.org/img/yt/AHxV5mTlp28.jpg)](https://www.youtube.com/watch?v=AHxV5mTlp28) [App Deployment and Automation - 20 tools and tips by Roy Osherove](https://pipelinedriven.org/library/2014/2/18/app-deployment-and-automation-20-tools-and-tips-by-roy-osherove) - [![](https://pipelinedriven.org/img/yt/rnGXSZ_jzDo.jpg)](https://www.youtube.com/watch?v=rnGXSZ_jzDo) [API Artifacts and Inter-Component Dependencies](https://pipelinedriven.org/library/2014/3/2/api-artifacts-and-inter-component-dependencies) - [![](https://pipelinedriven.org/img/yt/QyToLCtJm3M.jpg)](https://www.youtube.com/watch?v=QyToLCtJm3M) [Introduction to Beautiful Builds](https://pipelinedriven.org/library/2014/3/2/introduction-to-beautiful-builds) --- title: "Feature Toggles" url: "https://pipelinedriven.org/feature-toggle-frameworks-list.md" canonical: "https://pipelinedriven.org/feature-toggle-frameworks-list" tags: [] author: "Roy Osherove" description: "A list of feature toggle (feature flag, feature bit) frameworks for Java, Python, .NET, Ruby and more." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Feature Toggles Tools # Feature Toggle Framework List Also known as [Feature Bits and Feature Flags](http://martinfowler.com/bliki/FeatureToggle.html), here is a list of some of the frameworks you can find in the wild for your favorite language. [Here is a form to submit frameworks not listed here](https://docs.google.com/forms/d/1kH3YpF2Y8sR0oXhW7HE1YxSuUs_2XWahfaDhJtEC1EM/viewform) --- ## Feature Flags in Java 1. [Togglz](http://www.togglz.org/) 2. [FF4J](http://ff4j.org/) 3. [Fitchy](https://github.com/akomtur/fitchy) 4. [Flip](https://github.com/tacitknowledge/flip) ## Feature Flags in Python 1. [Gargoyle](https://github.com/disqus/gargoyle) 2. [Gutter](https://github.com/disqus/gutter) ## Feature Flags in .NET, C# 1. [FeatureSwitcher](https://github.com/mexx/FeatureSwitcher) 2. [NFeature](https://github.com/benaston/NFeature) 3. [FlipIt](https://github.com/timscott/flipit) 4. [FeatureToggleNET](https://github.com/jason-roberts/FeatureToggle) 5. [List on NUget](http://nugetmusthaves.com/Tag/toggle) 6. [FeatureBee](https://github.com/AutoScout24/FeatureBee) ## Feature Flags in Ruby and Ruby on Rails 1. [rollout](https://github.com/FetLife/rollout) 2. [feature\_flipper](https://github.com/qype/feature_flipper) 3. [flip](https://github.com/pda/flip) 4. [setler](https://github.com/ckdake/setler) 5. [switches](https://github.com/jpignata/switches) 6. [FluidFeatures](https://github.com/FluidFeatures) --- title: "Books by Roy Osherove" url: "https://pipelinedriven.org/books-by-roy-osherove.md" canonical: "https://pipelinedriven.org/books-by-roy-osherove" tags: [] author: "Roy Osherove" description: "Books by Roy Osherove: The Art of Unit Testing, Elastic Leadership and Enterprise DevOps." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Books by Roy Osherove Books - [![Cover: The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/img/c/aout2cover-f803c6.jpg)](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) ## [The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) The Art of Unit Testing guides you step by step from writing your first simple tests to developing robust test sets that are maintainable, readable, and trustworthy. You'll master the foundational ideas and quickly move to high-value subjects like mocks, stubs, and isolation, including frameworks You'll explore test patterns and organization, working with legacy code, and even "untestable" code. Along the way, you'll learn about integration testing and techniques and tools for testing databases and other technologies. [View on Amazon](http://amzn.to/2f1JRGo) - [![Cover: Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/img/c/elasticleadershipcover-3e1fbf.png)](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) ## [Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) Elastic leadership is a framework and philosophy that can help you as you manage day-to-day and long-term challenges and strive to create the elusive self-organizing team. It is about understanding that your leadership needs to change based on which phase you discover that your team is in. This book provides you with a set of values, techniques, and practices to use in your leadership role. [View on Amazon](http://amzn.to/2dWAVB0) - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-497482.jpg)](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) This book picks up where others have stopped: implementing DevOps at huge companies with numerous challenges relating to process, dependencies, monolithic systems and more. [Learn more](http://devops.osherove.com/) --- title: "Enterprise DevOps Books" url: "https://pipelinedriven.org/enterprise-devops-books.md" canonical: "https://pipelinedriven.org/enterprise-devops-books" tags: [] author: "Roy Osherove" description: "Roy Osherove's favourite books about DevOps, enterprise DevOps, continuous delivery and lean." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Enterprise DevOps Books Recommended reading The following list contains some of my favorite books about DevOps, Enterprise DevOps and DevOps related topics. I'm also the author of one of them, and I think it's important you see my book in the context of other books in the area.  If you think a book is missing from this list, [contact me](https://pipelinedriven.org/contact) and let me know. - [![Cover: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/img/c/cd-b5a069.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) ## [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) The "Bible" of CI/CD proponents, and one of my favorites. A great book covering almost any pattern and anti pattern you could think of, in a very deep manner. [View on Amazon](http://amzn.to/2f1FH1h) - [![Cover: The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/img/c/devopshandbook-f47030.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) ## [The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) A new one from 2016. Have not read this one yet, but it looks very promising based on the authors. [View on Amazon](http://amzn.to/2eRta3O) - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-8642f5.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) Written by yours truly. The book picks off where others have stopped: what happens when you try to apply DevOps principles in fortune 100 organizations? What lessons can be learned? The book is still in progress but can be purchased in beta form. [Learn more](http://devops.osherove.com/) - [![Cover: Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/img/c/ci-b18ab2.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) ## [Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) Another book I grew up on, talking about branching models, coding patterns and lots and lots of automation. This is but one of the practices you would be doing in a DevOps culture. [View on Amazon](http://amzn.to/2e6TMwu) - [![Cover: The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/img/c/phoenixproj-1806ac.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) ## [The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) A seminal book that helps explain what everyday IT organizations go through. A bit dated but still very relevant, it helps to explain lean concepts and the theory of constraints as applied to the IT business. Based off of the ideas in the book "The Goal" also mentioned on this web page. [View on Amazon](http://amzn.to/2fgnJgo) - [![Cover: The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/img/c/thegoal-7e0f02.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) ## [The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) A book that doesn't talk about IT, but is great to understand the lean toyota concepts, theory of constraints and much more. If you want to dig deeper into the roots of DevOps and Lean, this is a classic to dive into. [View on Amazon](http://amzn.to/2fgpzxR) --- title: "Beautiful Builds library" url: "https://pipelinedriven.org/library.md" canonical: "https://pipelinedriven.org/library" tags: [] author: "Roy Osherove" description: "The Beautiful Builds library: short videos by Roy Osherove on build patterns, continuous integration and deployment automation." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Beautiful Builds library Library # Beautiful Builds Short videos and the book on build patterns, continuous integration and deployment automation. - [![](https://pipelinedriven.org/img/yt/QuOOKrzrjfg.jpg)](https://www.youtube.com/watch?v=QuOOKrzrjfg) [Faster Feedback, Waves of Confidence, and Automatic Triggers](https://pipelinedriven.org/library/2014/2/18/faster-feedback-waves-of-confidence-and-automatic-triggers) - [![](https://pipelinedriven.org/img/yt/8jeIwGzuqJ8.jpg)](https://www.youtube.com/watch?v=8jeIwGzuqJ8) [Start by Shipping : Shipping Skeleton](https://pipelinedriven.org/library/2014/2/18/start-by-shipping-shipping-skeleton) - [![](https://pipelinedriven.org/img/c/title-page-244613.jpg)](https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book) [The Beautiful Builds Book](https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book) - [![](https://pipelinedriven.org/img/yt/mg5GzHbSZsw.jpg)](https://www.youtube.com/watch?v=mg5GzHbSZsw) [The Big Difference Betweeen Automated Builds and Continuous Integration](https://pipelinedriven.org/library/2014/2/18/the-big-difference-betweeen-automated-builds-and-continuous-integration) - [![](https://pipelinedriven.org/img/c/image-asset-3b82e3.jpg)](https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove) [Beautiful Builds - Roy Osherove](https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove) - [![](https://pipelinedriven.org/img/yt/AHxV5mTlp28.jpg)](https://www.youtube.com/watch?v=AHxV5mTlp28) [App Deployment and Automation - 20 tools and tips by Roy Osherove](https://pipelinedriven.org/library/2014/2/18/app-deployment-and-automation-20-tools-and-tips-by-roy-osherove) - [![](https://pipelinedriven.org/img/yt/rnGXSZ_jzDo.jpg)](https://www.youtube.com/watch?v=rnGXSZ_jzDo) [API Artifacts and Inter-Component Dependencies](https://pipelinedriven.org/library/2014/3/2/api-artifacts-and-inter-component-dependencies) - [![](https://pipelinedriven.org/img/yt/QyToLCtJm3M.jpg)](https://www.youtube.com/watch?v=QyToLCtJm3M) [Introduction to Beautiful Builds](https://pipelinedriven.org/library/2014/3/2/introduction-to-beautiful-builds) --- title: "DevOps & Enterprise DevOps Recommended Books" url: "https://pipelinedriven.org/enterprise-devops-books-list.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list" tags: [] author: "Roy Osherove" description: "Pipeline Driven by Roy Osherove: continuous delivery and DevOps best practices, build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # DevOps & Enterprise DevOps Recommended Books Books - [![Cover: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/img/c/cd-b5a069.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) ## [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) The "Bible" of CI/CD proponents, and one of my favorites. A great book covering almost any pattern and anti pattern you could think of, in a very deep manner. [View on Amazon](http://amzn.to/2f1FH1h) - [![Cover: The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/img/c/devopshandbook-f47030.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) ## [The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) A new one from 2016. Have not read this one yet, but it looks very promising based on the authors. [View on Amazon](http://amzn.to/2eRta3O) - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-8642f5.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) Written by yours truly. The book picks off where others have stopped: what happens when you try to apply DevOps principles in fortune 100 organizations? What lessons can be learned? The book is still in progress but can be purchased in beta form. [Learn more](http://devops.osherove.com/) - [![Cover: Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/img/c/ci-b18ab2.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) ## [Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) Another book I grew up on, talking about branching models, coding patterns and lots and lots of automation. This is but one of the practices you would be doing in a DevOps culture. [View on Amazon](http://amzn.to/2e6TMwu) - [![Cover: The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/img/c/phoenixproj-1806ac.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) ## [The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) A seminal book that helps explain what everyday IT organizations go through. A bit dated but still very relevant, it helps to explain lean concepts and the theory of constraints as applied to the IT business. Based off of the ideas in the book "The Goal" also mentioned on this web page. [View on Amazon](http://amzn.to/2fgnJgo) - [![Cover: The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/img/c/thegoal-7e0f02.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) ## [The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) A book that doesn't talk about IT, but is great to understand the lean toyota concepts, theory of constraints and much more. If you want to dig deeper into the roots of DevOps and Lean, this is a classic to dive into. [View on Amazon](http://amzn.to/2fgpzxR) --- title: "Books by Roy Osherove" url: "https://pipelinedriven.org/agility-devops-related-books.md" canonical: "https://pipelinedriven.org/agility-devops-related-books" tags: [] author: "Roy Osherove" description: "Pipeline Driven by Roy Osherove: continuous delivery and DevOps best practices, build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Books by Roy Osherove Books - [![Cover: The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/img/c/aout2cover-f803c6.jpg)](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) ## [The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) The Art of Unit Testing guides you step by step from writing your first simple tests to developing robust test sets that are maintainable, readable, and trustworthy. You'll master the foundational ideas and quickly move to high-value subjects like mocks, stubs, and isolation, including frameworks You'll explore test patterns and organization, working with legacy code, and even "untestable" code. Along the way, you'll learn about integration testing and techniques and tools for testing databases and other technologies. [View on Amazon](http://amzn.to/2f1JRGo) - [![Cover: Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/img/c/elasticleadershipcover-3e1fbf.png)](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) ## [Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) Elastic leadership is a framework and philosophy that can help you as you manage day-to-day and long-term challenges and strive to create the elusive self-organizing team. It is about understanding that your leadership needs to change based on which phase you discover that your team is in. This book provides you with a set of values, techniques, and practices to use in your leadership role. [View on Amazon](http://amzn.to/2dWAVB0) - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-497482.jpg)](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) This book picks up where others have stopped: implementing DevOps at huge companies with numerous challenges relating to process, dependencies, monolithic systems and more. [Learn more](http://devops.osherove.com/) --- title: "Contact" url: "https://pipelinedriven.org/contact.md" canonical: "https://pipelinedriven.org/contact" tags: [] author: "Roy Osherove" description: "Contact Roy Osherove about continuous delivery, enterprise DevOps, consulting or training." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Contact Contact # Get in touch Questions about continuous delivery, enterprise DevOps, consulting or training? Send Roy a message. Messages go through a short contact form (Google Forms). [Open the contact form (opens in a new tab)](https://forms.gle/jQSx5KoXvrVA1JCK7) --- title: "Faster Feedback, Waves of Confidence, and Automatic Triggers" url: "https://pipelinedriven.org/library/2014/2/18/faster-feedback-waves-of-confidence-and-automatic-triggers.md" canonical: "https://pipelinedriven.org/library/2014/2/18/faster-feedback-waves-of-confidence-and-automatic-triggers" published: "2014-02-18T11:13:34.083000+00:00" updated: "2014-02-18T11:13:34.083000+00:00" tags: [] author: "Roy Osherove" description: "Faster Feedback, Waves of Confidence, and Automatic Triggers: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Faster Feedback, Waves of Confidence, and Automatic Triggers Beautiful Builds library [![](https://pipelinedriven.org/img/yt/QuOOKrzrjfg.jpg)](https://www.youtube.com/watch?v=QuOOKrzrjfg) Beautiful Builds: Faster Feedback, Waves of Confidence, and Automatic Triggers · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "Start by Shipping : Shipping Skeleton" url: "https://pipelinedriven.org/library/2014/2/18/start-by-shipping-shipping-skeleton.md" canonical: "https://pipelinedriven.org/library/2014/2/18/start-by-shipping-shipping-skeleton" published: "2014-02-18T11:14:02.962000+00:00" updated: "2014-02-18T11:14:02.962000+00:00" tags: [] author: "Roy Osherove" description: "Start by Shipping : Shipping Skeleton: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Start by Shipping : Shipping Skeleton Beautiful Builds library [![](https://pipelinedriven.org/img/yt/8jeIwGzuqJ8.jpg)](https://www.youtube.com/watch?v=8jeIwGzuqJ8) Start by Shipping : Shipping Skeleton · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "The Beautiful Builds Book" url: "https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book.md" canonical: "https://pipelinedriven.org/library/2014/2/18/the-beautiful-builds-book" published: "2014-02-18T20:52:40.141000+00:00" updated: "2014-02-18T20:52:40.141000+00:00" tags: [] author: "Roy Osherove" description: "The Beautiful Builds Book: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The Beautiful Builds Book Beautiful Builds library ![The Beautiful Builds Book](https://pipelinedriven.org/img/c/title-page-244613.jpg) [Get the book on Leanpub](http://leanpub.com/build) [← Back to the library](https://pipelinedriven.org/library) --- title: "The Big Difference Betweeen Automated Builds and Continuous Integration" url: "https://pipelinedriven.org/library/2014/2/18/the-big-difference-betweeen-automated-builds-and-continuous-integration.md" canonical: "https://pipelinedriven.org/library/2014/2/18/the-big-difference-betweeen-automated-builds-and-continuous-integration" published: "2014-02-18T11:16:49.101000+00:00" updated: "2014-02-18T11:16:49.101000+00:00" tags: [] author: "Roy Osherove" description: "The Big Difference Betweeen Automated Builds and Continuous Integration: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The Big Difference Betweeen Automated Builds and Continuous Integration Beautiful Builds library [![](https://pipelinedriven.org/img/yt/mg5GzHbSZsw.jpg)](https://www.youtube.com/watch?v=mg5GzHbSZsw) The Big Difference Betweeen Automated Builds and Continuous Integration · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "Beautiful Builds - Roy Osherove" url: "https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove.md" canonical: "https://pipelinedriven.org/library/2014/2/18/beautiful-builds-roy-osherove" published: "2014-02-18T11:17:39.096000+00:00" updated: "2014-02-18T11:17:39.096000+00:00" tags: [] author: "Roy Osherove" description: "Beautiful Builds - Roy Osherove: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Beautiful Builds - Roy Osherove Beautiful Builds library ![Beautiful Builds - Roy Osherove](https://pipelinedriven.org/img/c/image-asset-3b82e3.jpg) This video is no longer available on YouTube. [← Back to the library](https://pipelinedriven.org/library) --- title: "App Deployment and Automation - 20 tools and tips by Roy Osherove" url: "https://pipelinedriven.org/library/2014/2/18/app-deployment-and-automation-20-tools-and-tips-by-roy-osherove.md" canonical: "https://pipelinedriven.org/library/2014/2/18/app-deployment-and-automation-20-tools-and-tips-by-roy-osherove" published: "2014-02-18T11:18:44.777000+00:00" updated: "2014-02-18T11:18:44.777000+00:00" tags: [] author: "Roy Osherove" description: "App Deployment and Automation - 20 tools and tips by Roy Osherove: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # App Deployment and Automation - 20 tools and tips by Roy Osherove Beautiful Builds library [![](https://pipelinedriven.org/img/yt/AHxV5mTlp28.jpg)](https://www.youtube.com/watch?v=AHxV5mTlp28) App Deployment and Automation - 20 tools and tips by Roy Osherove · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "API Artifacts and Inter-Component Dependencies" url: "https://pipelinedriven.org/library/2014/3/2/api-artifacts-and-inter-component-dependencies.md" canonical: "https://pipelinedriven.org/library/2014/3/2/api-artifacts-and-inter-component-dependencies" published: "2014-03-02T13:53:31.101000+00:00" updated: "2014-03-02T13:53:31.101000+00:00" tags: [] author: "Roy Osherove" description: "API Artifacts and Inter-Component Dependencies: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # API Artifacts and Inter-Component Dependencies Beautiful Builds library [![](https://pipelinedriven.org/img/yt/rnGXSZ_jzDo.jpg)](https://www.youtube.com/watch?v=rnGXSZ_jzDo) Beautiful Builds: API Artifacts and Inter-Component Dependencies · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "Introduction to Beautiful Builds" url: "https://pipelinedriven.org/library/2014/3/2/introduction-to-beautiful-builds.md" canonical: "https://pipelinedriven.org/library/2014/3/2/introduction-to-beautiful-builds" published: "2014-03-02T13:54:23.500000+00:00" updated: "2014-03-02T13:54:23.500000+00:00" tags: [] author: "Roy Osherove" description: "Introduction to Beautiful Builds: from the Beautiful Builds library by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Introduction to Beautiful Builds Beautiful Builds library [![](https://pipelinedriven.org/img/yt/QyToLCtJm3M.jpg)](https://www.youtube.com/watch?v=QyToLCtJm3M) Introduction to Beautiful Builds by Roy Osherove · YouTube [← Back to the library](https://pipelinedriven.org/library) --- title: "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation" url: "https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6" published: "2014-03-02T22:23:14.770000+00:00" updated: "2014-03-02T22:23:14.770000+00:00" tags: [] author: "Roy Osherove" description: "Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation Books - [![Cover: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/img/c/cd-b5a069.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) ## [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://pipelinedriven.org/enterprise-devops-books-list/svo6zadfkou0uv27kz6pxg0ev1twz6) The "Bible" of CI/CD proponents, and one of my favorites. A great book covering almost any pattern and anti pattern you could think of, in a very deep manner. [View on Amazon](http://amzn.to/2f1FH1h) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations" url: "https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5" published: "2016-10-29T06:10:12.837000+00:00" updated: "2016-10-29T06:10:12.837000+00:00" tags: [] author: "Roy Osherove" description: "The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations Books - [![Cover: The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/img/c/devopshandbook-f47030.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) ## [The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations](https://pipelinedriven.org/enterprise-devops-books-list/jkivba3nda04ezgyyuqgb0eua47yr5) A new one from 2016. Have not read this one yet, but it looks very promising based on the authors. [View on Amazon](http://amzn.to/2eRta3O) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "Enterprise DevOps: Patterns for Succeeding with DevOps at Scale" url: "https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6" published: "2016-10-29T06:53:16.748000+00:00" updated: "2016-10-29T06:53:16.748000+00:00" tags: [] author: "Roy Osherove" description: "Enterprise DevOps: Patterns for Succeeding with DevOps at Scale: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Enterprise DevOps: Patterns for Succeeding with DevOps at Scale Books - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-8642f5.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/enterprise-devops-books-list/g84kk40asdl2q2joacywae5vxsr4c6) Written by yours truly. The book picks off where others have stopped: what happens when you try to apply DevOps principles in fortune 100 organizations? What lessons can be learned? The book is still in progress but can be purchased in beta form. [Learn more](http://devops.osherove.com/) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "Continuous Integration: Improving Software Quality and Reducing Risk" url: "https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu" published: "2014-03-02T22:21:00.539000+00:00" updated: "2014-03-02T22:21:00.539000+00:00" tags: [] author: "Roy Osherove" description: "Continuous Integration: Improving Software Quality and Reducing Risk: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Continuous Integration: Improving Software Quality and Reducing Risk Books - [![Cover: Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/img/c/ci-b18ab2.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) ## [Continuous Integration: Improving Software Quality and Reducing Risk](https://pipelinedriven.org/enterprise-devops-books-list/9j1f9feuknuvl5ihh672gapn45tvvu) Another book I grew up on, talking about branching models, coding patterns and lots and lots of automation. This is but one of the practices you would be doing in a DevOps culture. [View on Amazon](http://amzn.to/2e6TMwu) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014" url: "https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc" published: "2016-10-29T06:56:33.004000+00:00" updated: "2016-10-29T06:56:33.004000+00:00" tags: [] author: "Roy Osherove" description: "The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014 Books - [![Cover: The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/img/c/phoenixproj-1806ac.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) ## [The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win Paperback – October 16, 2014](https://pipelinedriven.org/enterprise-devops-books-list/996b9gc085i16jh6htuycp8z0p2kqc) A seminal book that helps explain what everyday IT organizations go through. A bit dated but still very relevant, it helps to explain lean concepts and the theory of constraints as applied to the IT business. Based off of the ideas in the book "The Goal" also mentioned on this web page. [View on Amazon](http://amzn.to/2fgnJgo) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "The Goal: A Process of Ongoing Improvemen" url: "https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1.md" canonical: "https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1" published: "2016-10-29T06:59:10.473000+00:00" updated: "2016-10-29T06:59:10.473000+00:00" tags: [] author: "Roy Osherove" description: "The Goal: A Process of Ongoing Improvemen: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The Goal: A Process of Ongoing Improvemen Books - [![Cover: The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/img/c/thegoal-7e0f02.jpg)](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) ## [The Goal: A Process of Ongoing Improvemen](https://pipelinedriven.org/enterprise-devops-books-list/6xm8hxelahmh0b9h9wolmj7ilz7kh1) A book that doesn't talk about IT, but is great to understand the lean toyota concepts, theory of constraints and much more. If you want to dig deeper into the roots of DevOps and Lean, this is a classic to dive into. [View on Amazon](http://amzn.to/2fgpzxR) [← All books](https://pipelinedriven.org/enterprise-devops-books) --- title: "The Art of Unit Testing, 2nd Edition" url: "https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing.md" canonical: "https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing" published: "2016-10-29T07:13:26.678000+00:00" updated: "2016-10-29T07:13:26.678000+00:00" tags: [] author: "Roy Osherove" description: "The Art of Unit Testing, 2nd Edition: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # The Art of Unit Testing, 2nd Edition Books - [![Cover: The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/img/c/aout2cover-f803c6.jpg)](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) ## [The Art of Unit Testing, 2nd Edition](https://pipelinedriven.org/agility-devops-related-books/art-of-unit-testing) The Art of Unit Testing guides you step by step from writing your first simple tests to developing robust test sets that are maintainable, readable, and trustworthy. You'll master the foundational ideas and quickly move to high-value subjects like mocks, stubs, and isolation, including frameworks You'll explore test patterns and organization, working with legacy code, and even "untestable" code. Along the way, you'll learn about integration testing and techniques and tools for testing databases and other technologies. [View on Amazon](http://amzn.to/2f1JRGo) [← All books](https://pipelinedriven.org/books-by-roy-osherove) --- title: "Elastic Leadership: Growing Self Organizing Teams" url: "https://pipelinedriven.org/agility-devops-related-books/elastic-leadership.md" canonical: "https://pipelinedriven.org/agility-devops-related-books/elastic-leadership" published: "2016-10-29T07:17:38.561000+00:00" updated: "2016-10-29T07:17:38.561000+00:00" tags: [] author: "Roy Osherove" description: "Elastic Leadership: Growing Self Organizing Teams: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Elastic Leadership: Growing Self Organizing Teams Books - [![Cover: Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/img/c/elasticleadershipcover-3e1fbf.png)](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) ## [Elastic Leadership: Growing Self Organizing Teams](https://pipelinedriven.org/agility-devops-related-books/elastic-leadership) Elastic leadership is a framework and philosophy that can help you as you manage day-to-day and long-term challenges and strive to create the elusive self-organizing team. It is about understanding that your leadership needs to change based on which phase you discover that your team is in. This book provides you with a set of values, techniques, and practices to use in your leadership role. [View on Amazon](http://amzn.to/2dWAVB0) [← All books](https://pipelinedriven.org/books-by-roy-osherove) --- title: "Enterprise DevOps: Patterns for Succeeding with DevOps at Scale" url: "https://pipelinedriven.org/agility-devops-related-books/enterprise-devops.md" canonical: "https://pipelinedriven.org/agility-devops-related-books/enterprise-devops" published: "2016-10-29T07:21:28.837000+00:00" updated: "2016-10-29T07:21:28.837000+00:00" tags: [] author: "Roy Osherove" description: "Enterprise DevOps: Patterns for Succeeding with DevOps at Scale: a book recommended on Pipeline Driven by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Enterprise DevOps: Patterns for Succeeding with DevOps at Scale Books - [![Cover: Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/img/c/entdevops-497482.jpg)](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) ## [Enterprise DevOps: Patterns for Succeeding with DevOps at Scale](https://pipelinedriven.org/agility-devops-related-books/enterprise-devops) This book picks up where others have stopped: implementing DevOps at huge companies with numerous challenges relating to process, dependencies, monolithic systems and more. [Learn more](http://devops.osherove.com/) [← All books](https://pipelinedriven.org/books-by-roy-osherove)