Will Chat GPT and generative AI “replace” testing?

There is a lot of buzz within the software testing and development communities about Chat GPT, and the role of generative AI in testing.

Some of the opinion pieces, webinars and videos focus on the potentially beneficial applications of generative AI for testing speed and quality. However, many focus on how Chat GPT might impact a tester’s future job prospects and security. Some go as far as questioning whether Chat GPT will replace the role of manual testers and SDETs [1].

The popularity of this type of content is natural, given the anxiety associated with wavering job security in tech, and the natural hunger of testers to learn the latest tools and skills. Meanwhile, some organisations still view testing as a problematic cost centre. The appeal of automation and AI for these managers in turn becomes cost avoidance, while still going through the motions of “QA”.

How well-founded is the speculation that Chat GPT might “replace” testing, or, at least, replace testers? And, for senior management, how should you think about the role of AI-based automation in your organisation’s future QA investment?

A worry as old as automation itself

The view that newer technologies and newly skilled workers will render existing jobs redundant is not new. The term “luddite”, for instance, stems from a movement of textile works in 19^th Century England, who opposed textile machinery as a threat to the value of their highly-skilled work [2].

“Luddites” in the early 1800s opposed – and smashed – new textile machinery like looms. Original image: Wikimedia.

Within software testing, these same concerns emerged some 10+ years ago, when marketers presented test execution automation as a “silver bullet” for testing efficiencies and bloated testing budgets.

And now today, Chat GPT and generative AI brings the promise that it will automate the automation itself, automating the labours performed by SDETs and engineers. These toilsome tasks might include writing test scripts and Gherkin specifications, or sourcing test data and analysing requirements.

Let’s assume that ChatGPT and generative AI can automate these tasks. For the sake of this article, let’s park the sizeable challenges surrounding the data used to train ML/AI models, questions of test coverage, and issues associated with governance, bias, and more.

What room will be left for the testers and SDETs who have spent years crafting their skills and expertise, if intelligent automation can produce test automation at a fraction of the time and cost?

The testing discipline is going nowhere

Yet, automating test execution did not automate away testing or QA. In fact, it created a raft of new and complex processes, requiring new skills, tools, and dedicated roles like the SDET.

No matter how much testing has “automated”, there has always been more to automate and maintain. Testing exhaustively is more of an impossible goal today than it was a decade ago.

The average test team has automated just 15-20% of all tests [3], reflecting this ever-growing testing requirement. The total number of possible test scenarios that could be automated just keeps growing.

To understand this ever-growing “to do” list, we must consider the relationship between speed and complexity. We must consider how doing individual tasks faster tends to create more tasks to complete, however paradoxical this might at first appear.

Moore’s law and more to test

One appeal of automation is the time and cost efficiency by which it executes high-volumes of tasks. In testing, automation has been well suited to repetitive and recurring tasks like regression testing. Chat GPT and generative AI promise to expand the scope beyond scenarios pre-defined by a human, further reducing the labour associated with creating and running tests.

Yet, the efficiency gained through automation is not isolated to testing. If it were, testing could hope to close the gap on exhaustive test coverage. Instead, the testing requirement continues to grow in size and complexity.

Just as QA becomes faster in its individual processes, so does software design, development, and every constituent part of the SDLC. These individual efficiencies enable faster software delivery, making changes of greater functional size and logical complexity. This leaves less time to test more.

The number of logical combinations to test will continue to grow, as will the need to maintain and refactor ever-growing volumes of historical code and tests. No matter how much you optimise, your delivery processes will also grow in complexity:

As Moore’s law reminds us, this growing complexity of software systems will not slow down [4].

One benefit of automation is that we can offload this complexity and the associated workload to machines that are constantly growing in processing power. They can execute ever-more tasks, ever-faster. Yet, this efficiency in turn brings more complexity and, with it, more work to do.

The requirement to optimise and go faster will never go away, while the sum total of work in front of us will never in itself decrease.

How will Chat GPT sit alongside your testing?

It stands to reason, then, that Chat GPT and generative AI will not “replace” testing or remove the need to invest in QA. Like test execution automation before it, generative AI will provide a useful tool for moving faster.

Yet, there will always be a need for more work and at least a constant (if not greater) need for human input. Testers’ time might be applied less to repetitive tasks like scripting, but new processes will fill the void. Meanwhile, the creativity and critical thinking offered by testers will not diminish in value as these repetitive processes are automated; such creativity should be given greater freedom.

At the same time, your testers will have vital insight into how generative AI should be used at your organisation. Nothing is adopted overnight, and identifying the optimal applications of tools like Chat GPT will be an ongoing conversation, just as the testing community has continually explored and improved practices for getting the most out of test automation frameworks.

Lastly, as the volume of possible test scenarios grows, automation and AI will need a human steer in knowing where to target its efforts, even as we can increasingly use data to target test generation.

How can you use Chat GPT in your testing?

So, what can you do to get the most of Chat GPT in your testing?

If you are a tester or SDET, approach generative AI like you approached test automation tools and frameworks. Skill-up and learn new technologies, considering which are the best fit for your organisation. There are already a range of resources to learn how to leverage Chat GPT in testing, even if many are still quite speculative.

These new skills will not only help you remain competitive within an increasingly crowded job market; they will help you become a leader of beneficial change at your organisation.

If you are a QA manager or director, give your teams the time, space and investment needed to experiment and explore these new tools and techniques.

Change and improvement in testing practices are often blocked because teams are too busy pumping out deliverables using outdated techniques. Rather than hope that generative AI will provide a silver bullet to these growing inefficiencies, sacrifice a fraction of your teams’ available testing time to learn about new tools, and consider how they can best be applied. Remember that your solutions will lie in "engineering augmented by AI", just as test automation offered an accelerator – but not replacement – to Quality Engineering.

Lastly, if you would like to understand some of the ways that generative AI is being applied to test automation today, watch our live expert roundtable. In this roundtable, James Walker, Paul Gerrard and Jonathon Wright discuss the potential benefits and pitfalls of Chat GPT for testing today, while providing live demos of approaches that you can start putting into practice.

This is original content and was not generated using ChatGPT

References

[1] See, for example, Shailesh Kumar, “Can ChatGPT replace QA SDET ? End Of All Manual Testing Jobs in 2023 ? OpenAI Chatbot GPT vs Tester”.

[2] Learn more about the Luddites on Wikipedia.

[3] Capgemini, Sogeti (2022), The World Quality Report 2021-22, P. 23. Retrieved from https://www.capgemini.com/insights/research-library/world-quality-report-wqr-2021-22/ on December 12^th 2022.

[4] Learn more about More's Law on Wikipedia.