The two kingpins in this approach will be data and automation, working in tandem to convert insights about what needs testing into rigorous automated tests. But first, let’s consider why it remains so challenging to design, develop and test in-sprint.
20 years on, siloes remain THE challenge in software delivery
As the Agile Manifesto approaches 20 years old, the software delivery lifecycle remains riddled with silos. These silos not only create time-consuming miscommunication, they also amplify manual effort. Each time information moves from one silo to the next, it needs to be converted from one format to another:
These “information hops” delay releases and introduce defects as misinterpretation creeps in at every stage. Let’s now look at each silo in more detail.
Design
From a test and development perspective, gathering requirements in text-based documents and disparate diagrams is simply not fit for purpose. The fragmentary written user stories and documents are far removed from the precise logic that needs developing. Meanwhile, there is typically little or no formal dependency mapping between the text-based formats and static diagrams.
Development
Software designs therefore introduce bugs when translated into source code, in turn creating time-consuming rework. In fact, multiple studies estimate that requirements are responsible for over half of all defects,[i] while further research estimates that developers spend half their time fixing bugs.[ii] Design defects therefore take up a large chunk of the time that should be spent developing new functionality.
Testing
The static nature of requirements further increases manual effort in testing. “Flat” documents and diagrams are not ready built for automation, and often testers are forced to convert designs manually into test cases, data and scripts.
In addition to wasting time, these manual processes undermine quality. A simple system today will likely require thousands of tests before a release. Faced with informal and incomplete requirements, testers cannot systematically or automatically identify and create the tests that need executing before a release.
Manual test design instead focuses almost exclusively on “happy path” scenarios, over-testing these at the expense of scenarios most likely to cause bugs. Meanwhile, out-of-date and invalid tests pile up, creating test failures that push testing further behind releases.
Automation can help – but only if it extends beyond test execution
Automation can, of course, help accelerate many of these manual processes. However, “test automation” to date has focused near-exclusively on one task: executing tests. In so doing, it has introduced a raft of manual processes, while overlooking the key question of quality.
Test automation in many instances has introduced a new silo, along with all the time and effort associated with it. Manual processes introduced by test automation include copious test scripting, as well overwhelming script maintenance. Meanwhile, the speed and volume of test execution multiplies data provisioning bottlenecks, while out-of-date or inconsistent data leads to time-consuming test failures.
Automating test execution further does nothing to improve test quality, nor help with the identification of which tests to run. Test designers and automation engineers still face the challenge of having more system logic than they could ever execute in-sprint.
Prioritising the wrong tests not only wastes addition time in test scripting, it also exposes critical systems to damaging production defects. Test execution automation is therefore a critical component of in-sprint testing, but it is not a solution in itself.
“Data-driven” testing, but not as you know it
Fortunately, a solution is starting to present itself. It lies in data, and the ways that we can apply automation to data widely available today. This opens the door to prioritising tests accurately and automatically in-sprint, generating the tests needed before the next release.
The proliferation of (automated) technologies across DevOps toolchains has led to an associated proliferation of data. What’s more, this data is now being outputted in formats that can be captured and analysed.
If we combine this data with automated analysis and test generation, we can begin populating up-to-date test artefacts into the same tools from which the data was gathered. This then creates a closed feedback loop, collecting and analysing more data to drive rigorous testing in-sprint.
Let’s now look at the components of this “data-driven” approach to in-sprint testing.
Connectivity
The first prerequisite in this approach is connectivity between technologies across the application delivery lifecycle. If disparate tools cannot pass information between one another, silos will persist. There will not be enough data to analyse, and nor will it be possible to populate test suites across tools.
Connectivity between technologies is therefore paramount, and in-sprint technology must be built on open technologies. Fortunately, Robotic Process Automation and DevOps Orchestration tools can help to rapidly integrate disparate technologies.
The data produced by these disparate tools must furthermore be collected, feeding a baseline of data to harvest. Tools can then be applied to harvest insights from this single source of truth, informing testers of what needs testing in-sprint. Tools for analysis include, but are not limited to, AI and ML-based technologies.
At this point, data has been gathered and analysed, indicating what needs testing in-sprint. But, how can we build and execute the tests required to act on this information?
Rather than starting from the system requirements and working through a series of silos, Test Modeller analyses data from across the whole development ecosystem. It thereby informs and updates testing as the requirements or environment change, rather than playing a constant game of catch-up. In a nutshell, Test Modeller enables in-sprint test automation.