No! Continuous testing is not the same as test automation. Can you implement continuous testing without automation? That's also a no. There has been a lot of confusing talk about this lately, which is not helping practitioners understand what they should be doing in their projects to get to a continuous testing state.
So, what is continuous testing? Wikipedia (of course!) has a good intro to it based on literature and practitioners. Adam Auerbach is one that I particularly like because of his transformative mindset and practical approach based on his experience (and scars).
When I think of continuous testing, I think of it as the main (not the only) path to continuous quality. It is the stepping stone to achieving a higher quality product through Continuous Delivery. Once Continuous Testing has been reached, then it?s time to think about other ways that don?t involve running tests to detect defects or problems, but instead, prevent defects from ever being coded. As we all (should) know, preventing a problem is always better (and cheaper) than reacting to fix one.
The 4 Pillars of Continuous Testing
In the path to continuous testing, I?ve come to realize you actually have to build four pillars to support a continuous testing practice that is focused on improving overall product quality.
1. Code Quality
The first pillar is about building capabilities in the Continuous Delivery Pipeline to ensure code quality. That means helping developers build the best possible code. Some teams follow a coding style guide that describes the design principles and the code convention all the developers must follow for writing consistent code. Then when a change request comes in, anyone can read the code and make the edits, therefore minimizing maintenance.
Other teams use different strategies, such as weekly reviews where developers get together with a peer and go through the code to ensure they understood it. There are many other strategies, but the goal is to ensure the code is built correctly and tools are scanning that code and providing feedback on how good, secure and performant it is. Does that mean the applications being built will have high quality? No!
2. Pipeline Automation
The second pillar is about ensuring the good code that was built can flow across environments without manual intervention. If you?re asking why that matters, then you should take a look at this article about the modern software factory. In short, whenever there is manual intervention in a process, an opportunity to inject problems is created.
A person can inadvertently select incorrect libraries or configuration files in order to generate a build, which will then fail as tests start being run against it. This creates a (false) perception that the application quality is poor. Someone will have to troubleshoot and after some (valuable) time, they?ll find the application code is correct and it?s something wrong with the build.
The never-ending wash, rinse, and repeat cycle starts. Wasted time, effort, and cost lead to poor Quality deployment, which negatively impacts the perceived product quality. There is no room for waste in a Continuous Delivery pipeline.
3. Application Quality
By now you have ensured you have built an awesome code and it can be deployed to any environment in a ?zero touch? manner. This is when the third pillar comes into play. How do you ensure your awesome code is doing what it was originally intended to do? In other words, you built the things right, but did you build the right things?
That?s where requirements are key ? well, not just requirements, but unambiguous and testable requirements. Business analysts, developers, testers, and operations engineers (yes ? them, too!) must have a common understanding of what must be developed, tested, and released to users.
Otherwise, each of these personas will make their own interpretations to fill any gaps in the requirements, which will lead to developers creating a code that will not be fully tested and will be deployed differently than it was intended. The root cause of this misalignment in the understanding of a requirement is poorly specified requirements.
Doing a lot more testing and in an automated fashion is only required today because it prevents a snowball effect in this misalignment (i.e., we catch defects originated from this misalignment earlier and remediate faster). However, it is still slower and costlier than it would be if everyone had a common understanding and these defects weren?t even introduced in the code. We would test only once, tests would pass and we would move on. That means reduced TTV (Time to Value, a critical business metric), reduced effort (more free resources to build more code), and cost savings (to hire more developers).
4. Customer Experience
How many times have you created the perfect code, it was automatically and seamlessly deployed to development, testing, and staging environments, automated tests ran across all test levels and environments, then the code was automatically deployed to production, and?there were no positive business outcomes?
If you?ve been doing this long enough, you know unfortunately this is not uncommon. You may have done everything perfectly across the SDLC, but users don?t feel they?re getting a good customer experience.
There are a few strategies you can implement to prevent this from happening and some make more sense than others, depending on the nature of your business. A commonly adopted one is when teams release to production a fully instrumented application very frequently so they can slowly learn about users? behaviors as they go through those new or changed features. Teams learn from it and quickly adjust the code, release again and monitor. This is an ongoing activity that becomes part of the development lifecycle.
Regardless of the strategy you use, you should not underestimate how critical it is to consider the customer experience as one of the pillars in your continuous testing journey.
Continuous Testing Like a Boss
With the four pillars built, you now have a solid foundation to truly achieve a continuous testing state. Your team is no longer thinking about ?phases? in the SDLC. Requirements, development, testing, release ? these activities become second nature and are just part of the process to release a business idea or value to the user.
So, if you?re still thinking about automating your tests at the unit, component, system, and system integration (we can?t forget performance and security!) levels, you?re definitely on the right track. Just make sure to keep in mind the bigger picture and start tackling the four pillars to ensure your users will perceive quality as they use your product.
