Local Environments Have Become Impractical - Reasons Why QA Needs A New Approach

In today’s agile work environment, requirements change all the time, and you have to keep up. A QA engineer’s daily grind is retesting repeatedly, creating new builds, deployments, and checking the accuracy of fixed bugs.

But when you’re doing all these steps in a local environment, you’ll encounter one major inconvenience: the need to restore or update your testing environment along with every change from the production or development environment. In a technologically diversified team, this could cost you days of progress every month, among other disadvantages.

Here’s a round-up of all the pitfalls of local testing environments and the solution to improve the running of testing apps and software without wasting time and extra costs.

For QAs, a local testing environment is a software and hardware setup where the testing team executes test cases on a local machine. To simplify the unit testing, QA engineers use local environments because they have full control over them, and they can experiment without interrupting or affecting the work of other developers. 

Local testing environments can work out great for some QA engineers, as some of them have mentioned:

“As a tester, I’d recommend using a local test environment to quickly validate something before it gets deployed to a shared test environment. Faster feedback is the main benefit. There may be reasons not to test locally that are context-specific, but in general local testing is best practice.”

“A local test environment has its purpose. As long as you use it within its limitations, I’d never suggest that they shouldn’t be used. The main limitation, generally speaking, is that in various ways, it’s not the same as the production environment. If you understand how it is different, and understand how that limits the local environment, then there isn’t an issue.”

For other QAs, things aren’t necessarily black or white:

“First of all, I think local test environments are a good thing. Unless you are fortunate enough to have a dedicated remote service that can spin up instances of your application with ease on demand, in reality, the alternative is a shared environment like a staging or UAT server that will often have multiple parties interested in using it, causing all manner of potential data issues and priority over what work should be deployed.

However, we know that the reality of local environments is that you are often missing two key parts:

So often you compromise on testing due to a lack of this. You test something locally, and then when it goes live, all the real-world edge cases emerge that you couldn’t account for. This can be ameliorated by clever use of data seeding and mocking, but again, it’s rarely that straightforward and getting developer, and product buy-in to set this up and maintain requires persuasion and expertise.

These are the two reasons I wouldn’t recommend a local test environment, but on the other hand, having your own self-contained playground where you can reset data, switch branches, and generally experiment without interrupting anyone else’s workflow is a major plus.

It really comes down to the fact that there is no perfect environment, they all have their pros and cons. The ideal world would be a perfect replica of live production with all data and all integrations present, but we know that for most of us, that doesn’t happen.”

“Local testing has a time and a place. Quick code review checks are great. They aren’t meant to find all defects; they are meant to find risk areas and big major glaring issues. But big regression suites and integration tests are not suitable for local environments as they are generally large, expensive, and meant to be run on services that mirror what they do in production. Local environments aren’t constantly synced to use the latest revisions of code (unless the developer syncs). I have run across cases where I have found defects in a local version, but once the code merges in, the issues resolve themselves. I have also come across the exact opposite. It really is a situational and organizational decision. It’s not whether or not something is good or bad; it’s deciding when it is the right situation to use it (and whether it is the best option).”

Still, the trouble that most QA engineers face is the fact that mocking or replicating services in the local environment doesn’t translate to the live environment upon which your application will run during production. In other words, you’ll miss the chance to find potential flaws and not test applications in a realistic manner. Additionally, with every update or change on one type of environment, you’ll have to come and update your local machine too. This raises problems with always replicating different sections of environments.

Here’s what a few QA engineers have said about this:

“Local environments can’t test cross-origin situations where the non-local ones will have to retrieve data from other sources like API servers. The number of times a Dev told me ‘It worked great on my machine’ gives me a reason to never trust local testing. Of course, there are exceptions, but the test environment should be as close to the intended use environment as possible.”

“Two reasons, really. One is data... if you are testing something that retrieves data, you may believe that it is ‘fast enough’ for small amounts but fails utterly with large amounts that you would find in an integration environment. This also includes things like pagination or edge cases for large numbers of entries. Secondly, in a local environment, you can easily test logic, but you can’t test a lot of errors that will occur in a production/integration environment, like interfaces between services or permissions problems.”

“The only time that local test environments make sense to me is when you need to stand up a specific version of an application that is no longer running in a shared dev or shared QA environment. The shared environments typically have sanitized copies of live data (more test cases) and are running the latest code across the board, and might interact with 3rd party sandbox APIs.”

“If it works in a cloud beta environment, I wouldn’t recommend doing it locally; you’d end up doing just more work for no reason.”

“Test environments should match the live environments as much as possible. Which is quite often not the case in local test environments.”

These are only a few points of view, but they do speak from their own experience. To sum up, local testing environments aren’t bad; they just require more work and effort, often a luxury both QA engineers and companies can’t afford. The ideal way of solving the problem with replicating environments is for QA, product, and development teams to sync their environments in one place.

Let’s see what the other downsides of local testing environments are.

Like any job, a QA engineer’s role has its highs and lows. During their lows, though, QAs like to joke around and say that with every new test, they cross their fingers, pray and hope ‘everything will work out ok.’ The reality shows their prayers aren’t always answered.

And that’s why there are several reasons why QAs can deal with a series of troubles, particularly with local environments.

Configuring a local environment on your machine might sound like a piece of cake at first glance. However, things aren’t always limited to setting up a Selenium testing environment locally, running tests with it, and integrating it with various tools. 

Some local environments require creating test servers or the need to install special utilities or frameworks, and you might have to re-do all the installs if this environment crashes or disrupts for any reason.

It has become tricky and time-consuming for QA engineers to manually manage and recreate these environments.

As Mihai Chitic, Senior Software Developer at Bunnyshell argued:

When you’re running your tests locally, you have to settle for a limited number of services and even tests based on your CPU and memory. For instance, you can’t always detect every problem in one testing scenario with performance testing, but your resources limit the amount of testing you can perform. Sadly, you’ll face physical memory constraints or low-performing CPUs.

In a different use case like scalability testing, your workload may stay at the same level, but the CPUs and memory will change.

An experienced QA engineer, shared her thoughts specifically about this issue:

“For me, the main issue with local testing has been the memory of my laptop. Out of 12 GB of laptop RAM, I had to use 10 just to start the application. Apart from this, it was difficult to test in an additional environment. And every time someone added anything, I had to make another pull request. Yet, sometimes no one tells you they make a change, and that creates another bottleneck.”

Here’s another opinion from Ionut Chiris-Tocila, Business Intelligence Manager at Bunnyshell:

In some cases, you might be unable to use third-party services because they don’t offer a solution to run their service locally. While a possible workaround to this issue would be replicating the third-party’s API, it will never portray the reality 100%, and it also takes a lot of work to maintain it.

If you’re in the situation to run integration tests when using a local machine, you might face deployment bugs and failures that are hard or nearly impossible to detect. That makes you keep on running, believing the integration was made properly. Sooner or later, the undetected bugs or errors may cause network connectivity issues, configuration issues, boot crashes, etc.

You may have to run different tests, either for more than one app or for a different stage. Even if you’ve set them to be automatic, sometimes you have to perform serial testing, where each test from your local machine has to run one at a time. This serial testing will slow down the speed at which you can release a new product or app feature, reducing the overall velocity. When you work in a CI/CD workflow, you’d have to postpone entire builds until each test is complete 100%.

This is the kind of frustration that makes QAs find a workaround, but ones that make them double their work, as we found on from another shared opinion:

“We can only daydream about the perfect test environment; our UAT environment wasn’t kept up to date along with production, so there are all sorts of issues, missing functionality, and terrible performance. I’ve been trying to push for a new UAT environment that’s a replica of production, and everyone thinks it’s a good idea, but I can’t get anyone to set it up.

It stresses me out when I need to do performance testing or something on production after hours. I don’t want to crash the websites and no one to be available to restart them. But doing it on UAT would pretty much be useless. I also don’t have much confidence in only testing UAT, so I’ll end up retesting on production too.”

When over 50% of the world’s online traffic comes from mobile devices, mobile responsive tests are downright vital, including for mobile app development. But a local machine doesn’t offer reliable accountability for mobile and will rarely offer accurate test results for a behavior specific to mobile devices only. Sure, you can make a few tweaks, but even a Chrome device toolbar of a GUI has its limitations.

What you set up on your computer or machine may be completely different from your team mates. This situation creates the typical ‘worked on my PC’ frustration, leading to inconsistencies in testing, security, and performance. With several team members each working on their own local development environment, there will always be the back-and-forth that eats a lot of time, ensuring everyone has an identical stack version and is always updated with the latest releases. A remedy to inconsistency would be if everyone would agree to follow a set of clear guidelines. In reality, this doesn’t always happen.

Case in point, another verdict from a QA specialist:

“I have two reasons why I see local environments as a bottleneck:

Firstly, it can be difficult to test things that get triggered on real-time events. E.g., if you want to test that a notification is sent in 3 days, you may not be able to do that reliably on a local machine such as your laptop/desktop without making sure the app runs for 3 days straight. And even then, local machines will sometimes decide to restart unilaterally, power-saving settings on laptops can mess with some stuff, etc.

Secondly, I notice users (both QAs and developers) can find it difficult to work out where the line between ‘local environment problem’ and ‘actual problem’ exists. I’m decent(ish) at it because of my odd assortment of experience before I entered QA as a career, but not everyone has that. Sometimes, I’d rather just replicate it in an agreed-upon test environment than try to prove to a dev that a problem does exist outside of my local.”

The ‘single source of truth’ – the practice to keep all your testing information and results in a single place is hard to follow with a local testing environment. Even if the entire team agrees to work based on the same rules and framework, it’s not exactly easy to discover real-time changes of your fellow QAs. That means it will take time for all of you to be in sync and have a common reference, leading to low efficiency and productivity.

QA specialists are, clearly, not big fans of local testing environments. To ease their workflow and that of the entire IT team, they need automated provisioning of multiple servers, databases, storage, and environments. 

The swift solution to erase all the bottlenecks of working and testing on a local machine is called an Environments as a Service platform. That’s exactly what we do at Bunnyshell: allow you to deploy scalable workflows with automatic complex IT processes that include all the company’s environments in one place.

Unlike on-site environments, you can update and change all your environments from anywhere. Every participant can see all these changes without keeping track of and installing new versions of software. 

Our EaaS model is completely scalable and only requires you to use and pay only for your team’s resources.

Check out all the benefits of EaaS from Bunnyshell!

*Note: This article was inspired by a thread on Reddit where we asked QA specialists to share their thoughts and personal experience about local environments.