Testing in production has become more popular, as architectures become more complex and it becomes very challenging and resource-intensive for engineering orgs to release bug-free software quickly. Let’s look at seven reasons why you should start testing in production, and how to get started.
What is Testing in Production
Testing in production is software testing that takes place on live user traffic in real-time. Not meant to be a testing method that replaces shift left testing, production testing complements other types of testing like unit tests, integration tests or BDT. By implementing testing in production, developers can test in a real environment, to ensure their code will actually work for users. It is also an important enabler for continuous deployment.
To read more about testing in production and its challenges, read our blog “What You Need To Know About Testing in Production”. To learn how testing in production can help you improve software quality, keep reading.
Production Testing Benefits
Here are seven advantages developers and companies can gain by testing in prod:
1. Optimizing Testing Accuracy
Testing in production means you’re testing in the exact environment that will be used. This means you’re testing the real data, dependencies and configurations your customers depend on, and also all their edge cases. By testing this way, your test scripts cover your customers’ real requirements and identify any actual errors they encounter through their user experience. This makes these tests as accurate and efficient as could be.
2. Ensuring “Production-Ready” is Really Production-Ready
When testing in staging and development, many bugs can be caught, but not all of them. Software is still released to users with possible errors, due to various edge cases that weren’t taken into consideration, a staging environment that wasn’t an exact replica of production or lack of real data. By testing in production, developers and testers can ensure that the code that was pushed to production is really production-ready. No more “but it works on my machine”. With testing in production developers can ensure it works on the machine it’s supposed to work on, at the highest quality.
3. Achieving Agility & Continuous Deployment
Testing in production means you have to be light on your fight (or your keyboard) to ensure any actual errors in production result in a rollback or hotfixes so the users don’t have to deal with buggy software. This practice de facto forces you to monitor and deploy frequently and often, which is continuous deployment.
In addition, when you’re testing in production you can’t run the risk of many features breaking at once. So you’ll deploy features in bite-sized parts, and test and monitor how they perform in production. This will make you agile, enabling you to react quickly to changing requirements, implement customer feedback and deploy frequently.
4. Monitoring & Gathering Feedback
Testing in production is a quick and efficient way to get user feedback for your software. Production testing enables ongoing monitoring of which features work, which ones are used and how and which ones aren’t performing as required by your KPI baseline. These insights will provide you with observability into your product and users.
This observability can be used for immediate assurance that your product is working bug-free after a release, but also as a strategic way to help you optimize development and the roadmap. By releasing your software to some of your users through a beta release, testing in production enables gathering feedback to determine if you’re going for a full roll-out, or changing some features first and then redeploying to everyone.
5. Saving Resources
While testing in production does not replace testing in staging and development, it also relieves the burden of having to make staging a complete replica of production (see below) or of mocking. Maintaining staging and mocking can be a very resource-intensive and stressful task, especially if it’s the last chance to fix software for release. By enabling testing in production, engineering orgs can focus their resources on high quality development, optimal testing in staging to the best of their ability, and quick turnaround in case of bugs in production. This streamlining makes the process more efficient.
6. Complementing the Staging Environment
Your staging environment sucks. No, not really, but it can never be a perfect replica of production. This is because it lacks the real data, component structure, size and loads that your production environment is dealing with. If it is a perfectly synced replica of production, then you might be spending a lot of money and developer time, which you might not have. This means you could also be dealing with security issues, like replicating customer data in a place you’re not supposed to.
All this goes to say that staging is a good place to test your software, but it can’t be the last place. To ensure your code really works for users, it needs to be also verified in the wild, in real-life environments.
7. Verifying Payments
Out of the various scenarios that are hard to replicate in staging, payment processing is probably the most complex (and important) one. To test payments you will need real data, security guardrails and many dependencies, all of which are hard to replicate. By testing in production you can ensure the payments are actually being processed correctly.
How to Get Started with Testing in Production
There is no one way to test in production. Some ways are more extreme, like Chaos Engineering, while others provide a more incremental, stable way to test, like feature flags. Here’s a list of some of the most popular methods for testing in production:
Feature Flags - Turning different parts of the codebase on and off for users, to enable experimentation and quick rollback.
A/B Testing - Dividing the user base into two and giving each a different software version, to monitor which version performs better.
Canary Testing - Rolling out a certain part of the product to a limited number of users, to monitor performance.
Blue/Green Testing - Running two production environments, while deploying and making changes in one to enable quick rollback to the other.
Chaos Engineering - Randomly making infrastructure changes in production to ensure deployed code can perform in any situation.
As mentioned, testing in production compliments other types of testing. Loadmill automatically creates API tests for you, so you can ensure code gets to production bug-free. Get started with our free tier.