Testing in production has become more popular, as architectures become more complex and it becomes very challenging and resource-intensive for engineering orgs to release bug-free software quickly. Let's look at seven reasons why you should start testing in production, and how to get started.

What is Testing in Production

Testing in production is software testing that takes place on live user traffic in real-time. Not meant to be a testing method that replaces shift left testing, production testing complements other types of testing like unit tests, integration tests or BDT. By implementing testing in production, developers can test in a real environment, to ensure their code will actually work for users. It is also an important enabler for continuous deployment.

Production Testing Benefits

Here are seven advantages developers and companies can gain by testing in prod:

1. Optimizing Testing Accuracy

Testing in production means you're testing in the exact environment that will be used. This means you're testing the real data, dependencies and configurations your customers depend on, and also all their edge cases. By testing this way, your test scripts cover your customers' real requirements and identify any actual errors they encounter through their user experience. This makes these tests as accurate and efficient as could be.

2. Ensuring "Production-Ready" is Really Production-Ready

When testing in staging and development, many bugs can be caught, but not all of them. Software is still released to users with possible errors, due to various edge cases that weren't taken into consideration, a staging environment that wasn't an exact replica of production or lack of real data. By testing in production, developers and testers can ensure that the code that was pushed to production is really production-ready. No more "but it works on my machine".

3. Achieving Agility & Continuous Deployment

Testing in production means you have to be light on your fight (or your keyboard) to ensure any actual errors in production result in a rollback or hotfixes so the users don't have to deal with buggy software. This practice de facto forces you to monitor and deploy frequently and often, which is continuous deployment.

4. Monitoring & Gathering Feedback

Testing in production is a quick and efficient way to get user feedback for your software. Production testing enables ongoing monitoring of which features work, which ones are used and how and which ones aren't performing as required by your KPI baseline. These insights will provide you with observability into your product and users.

5. Saving Resources

While testing in production does not replace testing in staging and development, it also relieves the burden of having to make staging a complete replica of production or of mocking. Maintaining staging and mocking can be a very resource-intensive and stressful task.

6. Complementing the Staging Environment

Your staging environment sucks. No, not really, but it can never be a perfect replica of production. This is because it lacks the real data, component structure, size and loads that your production environment is dealing with.

7. Verifying Payments

Out of the various scenarios that are hard to replicate in staging, payment processing is probably the most complex (and important) one. To test payments you will need real data, security guardrails and many dependencies, all of which are hard to replicate. By testing in production you can ensure the payments are actually being processed correctly.

How to Get Started with Testing in Production

There is no one way to test in production. Some ways are more extreme, like Chaos Engineering, while others provide a more incremental, stable way to test, like feature flags. Here's a list of some of the most popular methods for testing in production:

• Feature Flags - Turning different parts of the codebase on and off for users, to enable experimentation and quick rollback.
• A/B Testing - Dividing the user base into two and giving each a different software version, to monitor which version performs better.
• Canary Testing - Rolling out a certain part of the product to a limited number of users, to monitor performance.
• Blue/Green Testing - Running two production environments, while deploying and making changes in one to enable quick rollback to the other.
• Chaos Engineering - Randomly making infrastructure changes in production to ensure deployed code can perform in any situation.

As mentioned, testing in production compliments other types of testing. Loadmill automatically creates API tests for you, so you can ensure code gets to production bug-free.