When teams are looking to transform, optimize, or cut costs in testing, where do they first look? More often than not, they follow the advice given by outside consultancies: “You haven’t got enough test automation”, “your peers have achieved xx% tests executed through automation”.
There are a number of problems with this advice:
So, what are some of the telltale signs that your testing efforts today aren’t as effective and efficient as they could be? Let’s explore some of the warning signs that the way you test today could do with some changes and a fresh outlook on your test approach:
These antipatterns lead to a poor return on your testing efforts. Testing becomes too costly, too time-consuming, and leads to questionable outcomes in terms of quality and your ability to implement change.
You probably recognize some of these challenges. Yet, if you are suffering from one of them, you likely have them all, but haven’t looked from the right angle. This is because all of these challenges stem from one common cause: Not setting testing up properly. They are all signs that you don’t have solid foundations in your test approach and strategy.
If you don’t consciously have a test approach today, then the chances are that you have unwittingly implemented the “upside-down test pyramid” in some form:
With this, all of your testing is likely more expensive, slower, and less reliable than it needs to be. Yet, this is unfortunately common in test teams across organizations today, regardless of their delivery approach or technology being tested.
The upside-down test pyramid will slow you down and cost you more the longer you persist with it.
So, what can you do? Take the first step in consciously implementing a test approach that matches the IT vision, technology and change methodology of your organization. And be lean about it.
Many organizations have been adopting smaller, iterative ways of working. Common goals include boosting agility and speed to market, while lowering cost and improving feedback (quality).
While these organizations have introduced plenty of new technologies and design patterns, testing has often remained large, cumbersome, and poorly suited to automation. As dev teams work so hard to keep design and build small, they are frustrated as test doesn’t (or wont) follow those same principles. This is fundamentally and evidently wrong.
If your team are trying to adopt agile, but your wider team struggle to carve out small iterations of delivery, you have more than a test problem. You have a “big ball of mud”. This will either be a tightly couple architecture, or a distributed monolith if you are in the cloud. You need to start decoupling.
Decoupling means establishing boundaries within your architecture. A good place to start is by understanding your interfaces, building models by analyzing how interfaces work today. BDD-style collaboration and monitoring production are both example methods of understanding how an interface works. Then, you should validate that these interfaces work in the way they were intended.
Once you know that interfaces are working as intended, you have an opportunity to rearchitect your big ball of mud. More relevant to this article, you also have an opportunity to change your inefficient test approach. You can start targeting the newly understood, observable and testable interfaces through API testing. And then, you are starting to turn that testing pyramid the right way up.
Yet, a good test approach involves more than doing different types of tests. Just as we want small, interactive delivery, we also want fast feedback. For that, we need to have a full stack test approach:
Full stack testing should not test everything every time. The architecture should act like a Lego kit for constructing test approaches, assembling “blast radiuses” that we need to test. Testing then becomes appropriately lean and targeted at the things we are changing.
In other words, we want to follow the architectural best practices that are used around lose coupling and isolation, so that we don’t end up unnecessarily testing interfaces that aren’t changing and so pose zero risk. Targeted tests should live and run alongside the component they test. They then provide immediate and specifically directed feedback on what’s changed, and which might have gone wrong. If those best practices don’t exist, we can establish them through our test approach.
The diagram and table below indicates the broadened scope that testing should consider during a more full stack approach. You could easily go further than this to make testing more agile. Have a think for yourself about where you might add further separation:
Want to learn more about isolating blast radiuses for testing your systems? Join my next webinar.
Being able to isolate creases in your system architecture and use different types of testing in turn replaces the traditional phases of testing from waterfalled projects of old, and those that carried into newly formed agile teams.
An effective test approach in this context no longer talks about System Test or System Integration Test as phases. Instead, you have types of tests that you execute continually as part of your pipeline, and these tests are harmonized and targeted to the area of the system you are changing, including testing platforms.
Testing has been traditionally focused on functionality, but being able to isolate change involves getting closer to the system stack where the change is happening. This is not just API testing; it involves testing things like the platform and moving towards compliance as code.
Testing continuously at the platform level will be especially important as organizations transition to cloud, risking challenges in test environment builds and production outages due to manual and ambiguous config changes. We’ve all been there, yet many test approaches still don’t include such test types.
The problem is, a fully automated test pack of your user interface is going to fail as a non-starter if the environment you run it in is incorrectly configured. Check out the bugs you raised in your last release: Are environment problems ranking as a top cause of defects? Are you looking for these problems in your testing? Or you are finding these things far to late in your testing efforts, leading to substantial delays as your teams must spend time triaging the problems?
If this sounds familiar, your testing isn’t targeted sufficiently. Taking a lean approach to your test approach will do more than just improve the effectiveness and efficiency of your testing; it will make your whole development effort more effective. It will also create a change in mindset to help your team collaborate and strive for continual improvement.
Fast feedback is the key thing your test approach must provide within newer development practices. Automating test execution alone misses the mark of providing this critical capability. Before you automate, take a step back to revise your test approach. Go slower to go faster…..
To learn more about continuously targeting your testing across your whole system architecture, join my upcoming webinar: “Isolating Blast Radiuses For Testing: How to not blow up your DevOps world!”