The four biggest issues with having static environments

An environment is a set of one or more servers, configured to host the application we are developing, and with the application already installed on them, available for either manual or automated testing.

What are static environments?

Static environments are environments that are long lived. We do not destroy the environment, but instead keep loading it with the latest and greatest version of the application.

For example, we might have the following static environments:

“DEV”, “STAGE” and “PROD”.

Each one is used for a different purpose and by different crowds. What’s common about them is that they are all long lived (sometimes for months or years), and this creates several issues for the organization:

1.     Environment Rot: As time goes by, the application is continuously installed on the environments and configuration is done on them (manually). This creates an ongoing flux of changes to each environment that leads to several problems

a.     Inconsistency between environments (false positives or negatives)

  • Any deployment or tests results you get in one environment may not reflect what you actually get in production. For example, tests that pass in “Model” could be passing due to a specific configuration in MODEL that does not exist in other environments, meaning we’d get a false positive.
  • Bugs that happen in one of the environments might not happen in production, which is a false negative.

b.     Inability to reproduce issues between environments

If a bug is manifested in one environment but cannot be reproduced in another, due to the fact that the environments are different “pets”

2.    Long and costly maintenance times

Because environments are treated as “pets” (i.e you name them, you treat them when they are sick, each one is a unique snowflake that has its ups and downs), it takes a lot of time and manual, error prone activities, to maintain the environment and bring it up if it crashes.

This causes a delay whenever a team needs to test the product on an environment.

3.    Queuing

Because of teams waiting to deploy to an environment, and because there is a limited number of these environments (because they are costly to set up, maintain and pay for to keep running 24-7), queues start to form as teams await environments to become available.

This queueing can also be caused because multiple teams are expecting to use the same environment, and so each team waits for other teams to finish working with the environment before they can start working.

This causes release delays.

4.    Waste of money

 Static environments usually run 24-7, and in a private or public cloud scenario this might mean paying per hour per machine or VM instance. However, most environments are only used during work hours, which means in many organizations up to 16 hours of idle paid time.

Solution: 

In the next post I'll cover ephemeral environments and how they solve the issues mentioned here.