ARK 4: Environments - Asking Fore a Friend!

Simon Frazer

FORE-WORD

This blog follows on closely from Brian’s blog on Hardware, and Andrew’s on SDLC. Check those out in advance for further context to go with this blog.

Now those with a keen eye might have noticed the use of the word “fore” in the blog title and section heading. A cheap clickbaity pun, but as a keen golfer, let me start with a golf-related analogy – bear with me! Let’s consider a typical week’s work for everybody’s favourite golfer, Rory McIlroy.

The week might begin by spending hours at the practice range, putting green and the gym, developing his skills, and ensuring he’s in peak condition.
He’ll then head to the course and test himself during a practice round.
After all the practice and testing, it’s time to put it all into action in a competition! (unless it’s The Masters in which case it’s time for a nice walk for 2 days)

So how does this (very loosely) relate to software development, and specifically, environments? Well, when it comes to typical software development lifecycles (SDLC), it turns out there are a lot of similarities!

Practice range = Developers will have to write code to deliver a new feature, architecture, bug fix, etc, and they will do so first in the safety of a development environment, where they can experiment with different solutions, without worrying too much about getting it wrong or the impact of doing so.
Practice round = Once the developer is happy with his new code, it will move onwards to environments where it can be thoroughly tested.
Competition time = Once testing is complete, it can finally arrive in the production environment, ready to be used by users, or as part of an application.

In short, environments are defined by the intentions of the activities.

Typical Environments

Ok, that’s the end of the golf analogies – they’re out-of-bounds now.

In the most simplistic and common setup, you could typically have three environments:

Development: A safe place for developers to write code without fear of having any negative effects on the system and other areas of code, as nobody will be using the environment for meaningful business work.
Test: An environment to keep as ‘clean’ as possible to be able to test, manually or automated, new code that has been developed in the development environment and migrated to test.
Production: This contains all the code used by the users/application to deliver its purpose for the business.

The first thing to note here is that typically, terminology is not consistent across the industry. Sandwiched between development and production, which hopefully are universally understood, can be any combination of a variety of environment names, where the meaning and intention can vary from business to business. That being said, usually, the name of the environment is a giveaway for its intentions.

With this in mind, for a three-environment setup like this, you might hear Test interchanged with QA, Pre-Prod, or Staging, to name a few. Again, in a simple three-environment setup, it doesn’t really matter what name is used, because you will know the intention – to test changes before being released to production! However, it is important to note the difference in these environment names.

QA: A form of test environment, usually used and managed by a team of QA’s, business analysts, and testers, to help test code. Often it may have production data (or a small subset of) available for realistic testing.
Pre-Prod or Staging: The names are often used interchangeably. Typically used as an environment to hold new changes prior to a production release, with limited access for some to be used as testers. Maybe more commonly known by the term ‘beta’ env. Sometimes it is used more simply as a test environment and given one of these names.
DR: Disaster Recovery. Often treated as a separate environment, but also can be just an extension of the production environment. The name is a giveaway – it intends to help when the production environment has issues and/or for backup and business continuity purposes.

Environment Considerations

We see businesses with any combination of the above environments, but at this stage, you might be thinking, “How would I decide on my own environment setup?”

There are lots of things to consider when deciding on the right environment setup for you, your team, your application, and your business.

Requirements

Let’s start at the start – requirements!

For anyone needing a multi-environment SDLC, you’re going to have to consider the requirements at each stage of the process to be able to understand what environment setup could work. At every stage of the process, there will be decisions to be made that cannot be made without knowing what the requirements and intents of each environment are.

Some examples include:

Performance and capacity of the environment (another plug for Brian’s blog on Hardware)
Costs or budgeting constraints for the environment
Environment setup in relation to SDLC (and another signpost to Andrew’s blog on SDLC)

That’s all I’ll mention on requirements at this stage, as they will form a part of the conversation as we continue.

SDLC

One of the key considerations when it comes to environment setup, is your software development lifecycle. In cases where this is already established, your environment setup should be easily defined to work with this process. In other cases, you may be starting from scratch, and so in considering what environment setup you need, you will likely have your SDLC design as a key requirement.

One of the most important aspects of the SLDC will be the change management process. This is, the process you must follow to obtain approval for a release to go ahead. Environments can often play a key role in obtaining this sign-off, as thorough testing throughout well-maintained and constrained environments will only support your change request. This is a process that is often internally and externally audited, so it’s an important part to get right.

The next consideration regarding environments and SDLC involves people. In some cases where resources are more heavily constrained, or the application is small, the developers might be solely responsible for their testing. In which case, they might have a simple three env (dev, test, prod) setup. In larger teams, including BA’s, QA’s, and/or testers, more environments might be at play, with each group responsible for the flow and testing of code at each stage.

The Application

The application(s) that are being used and developed can often play a part in environment setup. If it is a straightforward application, maybe with a few (or less) people/teams responsible, then a simple, common, three env setup might suffice. As size and complexity grow, normally, so do the requirements of the environments and so a much more customized setup is required. Often the setup may be defined and agreed across teams and applications so that it aligns the end-to-end SDLC process for all.

Servers

Often, when considering environment setup, the setup and spec of servers are one of the main talking points, for obvious reasons.

You will want to consider the requirements in each environment, particularly when it comes to performance and storage. Often, the requirements for servers in each of the environments will vary considerably.

The first consideration comes at the start of the flow, and understanding the needs for a development environment. As a general rule, performance and storage, and any other server specs, will be less important in this environment, and so will be considerably underwhelming in comparison to other environments.

At the opposite end of the flow, the production environment will have a clear specification. This will be based on things like data storage, number of users, expected CPU load, peak times etc. Given the importance of this environment, this will be the highest-performing or most capable environment.

In between these, any other environments will probably sit somewhere in the middle in terms of specs.

When it comes to server commonalities across environments, operating systems, and general server settings will be consistent throughout.

Note that in some cases, depending on SDLC or the application, environments may run on local hardware rather than shared resource servers. For example, a development environment that runs on local machines and can filter into version control systems such as Git, or applications that are shipped as a package to be installed, such as a mobile phone app.

Other requirements placed on server setup to match environment needs may include asking questions such as:

Location – Are there restrictions or requirements on where the server should be located? For example, in cases where there is a significant focus on low latency, it may be a requirement for the production environment to be co-located to a data center.
Environment Links – Should servers & environments have links in order to be able to communicate, for example, to share data?
External connectivity – Do any of the environments require access to external systems/data/tools, for example, to query another database? If so, is this necessary in dev, or is some dummy data sufficient?
Multi-Server Environments – Are the applications significantly large or complex that they require multiple servers, for example, to load balance? If so, and assuming that isn’t the case in test environments, how can the test environment perform its duty of testing like a production environment?
Access – How is access and permissions to each environment managed? Are there other systems to interface with for this authentication? If so, is this required for dev/test environments?
Support & Maintenance – Who supports and maintains each environment? How can it be ensured that each environment is suitably maintained so that they are aligned as required?

As a final note on servers, it is more common nowadays to make use of cloud computing, so there can be different considerations on environment setup when it comes to hardware resources. Specifically, referring back to my earlier point about performance and storage, cloud computing makes it easy for these attributes to be dynamic and essentially have a pay-as-you-go service, so the consideration here might be what is the upper limit in terms of costs/billing each month for usage. Brian’s blog I signposted earlier discusses this in more detail.

Data Availability

One of the most common complaints when it comes to environments is a lack of data, or at the very least, a lack of data quality.

How can you develop without data? How can you test without good data?

Unfortunately, there is no common solution to this, and so this is something that must be considered as part of the environment setup. Any possible solution will often have to be cleared with data security teams, particularly if you need to take production data into a test environment. Often development environments suffer from having to make do with old or dummy data.

Business Continuity

Depending on the importance and nature of the application, a ‘backup plan’ may need to be factored into any environment setup. Often, businesses will have a DR (disaster recovery) environment, with the general premise that it should mirror the main production environment both in functionality, performance capabilities, and data – essentially duplicating your production environment. Naturally, there are costs that come with that, and so there are a lot of variations as to what a DR environment can look like.

A hot-hot setup describes the costly architecture of having both your production and DR environments co-existing, simultaneously. The benefits are mostly clear – there should be no disruption to the business if the production environment has an issue. Often teams will use the hot-hot setup to their advantage – sometimes for load balancing in peak periods, sometimes to test hot patch releases, and often as a way to move production data around.

Alternatively, a hot-cold setup might refer to a DR environment that is sitting dormant, waiting to kick into life if required during a production issue. The idea is that it will have access to the same data and features and so should be able to support the production use case. However, if startup time is greater than 0, then you are looking at a period of time where you may not be able to do so.

The environments would often not be located in the same datacentre, or at least within different networks, with the idea being that a network or datacentre outage wouldn’t wipe out both production and DR.

It may also be that DR is not as performant as production, where it can support some of the use cases but not all, with the aim of saving cost.

All of this must be considered when deciding on a DR environment.

So More or Less Environments?

So with all of these points brought into consideration, you might wonder – Is it better to have more or less environments? Good question.

One side of the argument says that having more environments creates more work, and leads to more problems. But a more structured, lengthier process should, in theory, allow for more opportunities to thoroughly test and catch bugs.

On the other side of the argument, fewer environments mean fewer opportunities to catch bugs before a production release. It also makes it difficult to establish a fully production-like test environment, given that with maybe only one environment between dev and production, that test environment has to be flexible to suit a lot of requirements. This setup, however, should be an easier setup to manage and maintain.

Ultimately, we could spend a long time looking at the pros and cons. But, given everything mentioned before, your own specific requirements will define what is important, to help you achieve the pros you need, and put up with the cons that this causes.

Watch out for the next blog in our ARK series which will cover disaster recovery and business continuity in detail.